Применение диффузионных вероятностных моделей к системам синтеза речи и методы ускорения их работы тема диссертации и автореферата по ВАК РФ 00.00.00, кандидат наук Попов Вадим Сергеевич

• Попов Вадим Сергеевич
• кандидат наук
• 2026

Оглавление диссертации кандидат наук Попов Вадим Сергеевич

Contents

Introduction

1 Non-autoregressive text-to-speech synthesis with diffusion models

1.1 Diffusion probabilistic modeling

1.1.1 MR-VP forward diffusion

1.1.2 MR-VP reverse diffusion

1.1.3 MR-VP loss function

1.2 Grad-TTS

1.2.1 Inference

1.2.2 Training

1.2.3 Model architecture

1.3 TTS experiments

1.3.1 Subjective evaluation

1.3.2 Objective evaluation

1.3.3 Efficiency estimation

1.4 Additional details

2 Voice conversion with accelerated diffusion models

2.1 Voice conversion model Diff-VC

2.2 Maximum likelihood SDE solver

2.3 Asymptotics of ML solver coefficients

2.4 Proof of Theorem

2.5 ML solver coefficients for various diffusion types

2.5.1 MR-VP DPM

2.5.2 Sub-VP DPM

2.5.3 VE DPM

2.6 Experiments

2.6.1 Any-to-any voice conversion

2.6.2 Maximum likelihood sampling

3 Voice conversion under optimal transport assumption

3.1 Optimal transport

3.2 MR-VP DDIM solver

3.3 Main experiments: voice conversion

3.4 Additional experiments: timbre transfer

Conclusion

References

List of Tables

List of Figures

A Appendix. Translation into Russian

Введение диссертации (часть автореферата) на тему «Применение диффузионных вероятностных моделей к системам синтеза речи и методы ускорения их работы»

Introduction

Topic of the thesis

Diffusion probabilistic modeling [1, 2] is an approach to generative modeling employing stochastic calculus [3], score matching techniques [4] and deep learning [5]. Diffusion probabilistic models (DPMs) have recently shown impressive results in generating data belonging to various domains including natural images [6, 7], video [8], graphs [9], molecules [10] and chirographic data [11] to name but a few. This work studies applicability of DPMs to audio domain and focuses on two tasks: text-to-speech synthesis and voice conversion.

In this work we introduce a generalized version of a conventional variance preserving diffusion proposed in the seminal paper on diffusion modeling [2], develop techniques necessary to train it and sample from it, and build a text-to-speech model based on the generalized diffusion. Then we make proper amendments to the text-to-speech model to enable zero-shot voice conversion. Because of iterative nature of sampling from diffusion models and complexity of the task of synthesizing unseen voices in a zero-shot manner the resulting voice conversion model has very low inference speed when standard sampling methods are used. To mitigate this issue, we propose maximum likelihood sampling method allowing to accelerate inference and provide theoretical results justifying the proposed method. Finally, this work discovers advantages of the style transfer method relying on the optimal transport assumption when applied to the voice conversion model.

Relevance of the work

Speech synthesis corresponds to a variety of tasks aiming at producing intelligible naturalsounding speech conditioned on some input. For text-to-speech models input is just a text; if additionally some reference voice sample is given as input, then a model synthesizing speech with this reference voice is said to solve voice cloning task; if no text is given and speech content is to be inferred from some source speech utterance and pronounced with the reference voice also given as input, then we deal with voice conversion; handling various speech modalities like singing or whisper as well as other relevant input types (e.g. video with people talking) also relates to speech synthesis. The scope of this work includes text-to-speech (TTS) and voice conversion (VC) tasks.

Speech synthesis algorithms can be divided into two main classes: concatenative speech synthesis [12] and statistical parametric speech synthesis [13]. Algorithms belonging to the

former class suggest producing speech by concatenating pre-recorded speech units such as diphones (pairs of consecutive phones) or triphones (triplets of consecutive phones). Despite fairly good results of a properly designed and trained concatenative speech synthesis system, it lacks flexibility since its performance is highly dependent on size and quality of the pre-recorded speech unit database [14]. Statistical parametric speech synthesis (SPSS) makes use of statistical models of speech trained with machine learning algorithms on human speech corpora. The process of generating speech with SPSS systems doesn't depend on pre-recorded phones making such systems a better choice especially when capturing different voices (including those unseen during training), different emotions, and other speech variation is important.

Typical SPSS systems solving the text-to-speech task are composed of three main parts:

• Text analysis module. The input text should first be processed to provide a standardized and phonetically plausible information for the subsequent generative modules. For example, input pieces of text "4 cats" and "four cats" must be mapped into the same sequence of phonetically meaningful units such as phonemes "fo: k^ts" belonging to the International Phonetic Alphabet (IPA). Standardizing text input (e.g. processing numbers, dates, time, money units, phone numbers, e-mail addresses, acronyms, etc.) is called text normalization; it is typically followed by phonemization whose result is then passed to the next module.

• Feature generation module. Instead of directly predicting raw waveform from input phonemes it can be easier to produce some intermediate acoustic features (e.g. compressed time-frequency features such as mel-spectrogram [15]) first. The reason is that raw waveform itself is redundant in the sense that there are lots of waveforms corresponding to perceptually indistinguishable speech (e.g. waveforms that differ by phase shift or amplitude characteristics of recording devices). Furthermore, high-quality speech is sampled at rates typically no less than 16kHz, i.e. 16000 samples per second, and the neighbouring samples appear to be highly correlated [16], while intermediate acoustic features sufficient to produce high-quality speech are sampled at much lower rates (typically only hundreds of samples per second) and they are less dependent on each other which makes it easier to synthesize them. Module trained to generate such features from input phonemes is called an acoustic model.

• Waveform generation module. Usually, a vocoder serves as this final module converting acoustic features into raw waveform corresponding to the input sequence of phonemes. Vocoder [17] is an algorithm that encodes speech waveform into a

compact time-dependent representation that can be efficiently decoded back into speech waveform with minimal loss in perceptual quality. So, apart from being a speech codec, a vocoder can be used as a part of a TTS system if an acoustic model is designed to produce the same compact speech representation as that utilized by this vocoder.

While first successful SPSS systems used classic machine learning models (e.g. hidden Markov models and Gaussian mixture models [18, 19]) as vocoders and acoustic models and employed speech signal processing techniques in their design, rapid development of deep learning enabled fully data-driven approach based on deep neural networks [20, 21]. This work also utilizes the mentioned approach to build the second part of the speech generation pipeline described above, i.e. it deals with deep neural acoustic models. For the text-to-speech task we consider models generating mel-spectrogram speech representation from input text whereas for the voice conversion task we consider generating mel-spectrograms corresponding to the speech content contained in the source mel-spectrogram. In the former case, generated speech corresponds to a single voice from the training dataset (i.e. we consider the so-called single speaker TTS) while in the latter case it corresponds to a voice extracted from a reference speech sample given to the voice conversion system as input along with the source mel-spectrogram. If both source and reference voices do not necessarily belong to the training dataset, voice conversion is said to operate in a zero-shot manner, and it is this mode that we focus on.

Generative neural networks used for acoustic feature generation fall into two main categories: autoregressive [21] and non-autoregressive [22, 23]. Despite being quite a natural choice because of sequential nature of human speech, autoregressive models suffer from several problems. Autoregressive feature generators usually rely on attention mechanism [24] whose failure can lead to mispronunciation issues like word or phone skipping and repetition. Also, autoregressive synthesis is prone to accumulation of errors in previously generated words or phones. Moreover, it is not efficient in terms of inference speed. To circumvent these drawbacks, non-autoregressive generative architectures based on variational autoencoders [25] and normalizing flows [26] have been proposed. In this work we consider another kind of generative models, namely DPMs [2], as acoustic models. Such a choice is motivated by the state-of-the-art results diffusion models show in generating data belonging to other continuous domains (e.g. natural images [6] and video [8]).

Classic diffusion models operate on data domain and do not have latent space in contrast with the already mentioned variational autoencoders, or, for example, generative

adversarial networks [27]. This feature allows for stable training and makes components of the systems based on DPMs more interpretable. In this work we modify one of the most successful non-autoregressive acoustic models for the text-to-speech task called Glow-TTS [22] by replacing normalizing flows generative engine with the one based on diffusion models resulting in the acoustic model we call Grad-TTS. Thus, Grad-TTS model proposed in this work consists of three parts: (i) text encoder, a module that encodes input text sequence; (ii) duration predictor, a module performing alignment between encoded text sequence and target speech frames; (iii) diffusion-based decoder, a module converting encoded text to mel-spectrogram. While in Glow-TTS encoded text sequence expanded by the duration predictor does not have any clear interpretation (it is just some latent sequence transformed to target mel-spectrogram by means of the decoder via normalizing flows), in Grad-TTS this expanded sequence approximates target mel-spectrogram, and this approximation is then refined by means of the diffusion-based decoder. Such a scheme allows us to think of encoded text sequences expanded by the Grad-TTS duration predictor as of rough mel-spectrograms. Hence, in contrast with Glow-TTS they now have a clear interpretation. Apart from delivering better speech generation quality due to superiority of DPMs in generative tasks in continuous domains over other generative frameworks, the choice of a DPM as acoustic model also improves prosody to a large extent. The reason is perhaps in the fact that it is easier to learn correct and accurate alignment between text input and generic acoustic features (e.g. mel-spectrogram as in Grad-TTS), albeit rough, than that between text and some latent acoustic features (as in Glow-TTS).

As far as the voice conversion task is concerned, the duration predictor in Grad-TTS architecture is no longer necessary - lengths of source and generated mel-spectrograms must be the same by the task formulation, and no expansion is necessary in contrast with the TTS task where length of input text sequence is not the same as the one of generated mel-spectrogram in time axis. Thus, Grad-TTS modification capable of voice conversion is a classic encoder-decoder voice conversion architecture [28, 29]: it consists of (i) source mel-spectrogram encoder, and (ii) diffusion-based decoder conditioned on reference speech utterance. As in all encoder-decoder voice conversion models, one of the most important and difficult tasks is to disentangle speech content of the source mel-spectrogram from speaker identity information contained in it. In this work "average" mel-spectrogram is proposed as the way to perform such disentanglement: the encoder is trained to convert source mel features into "average" ones, i.e. aggregated across all speakers in the training dataset in a phoneme-wise manner. In other words, encoded source mel-spectrogram corresponds to the speech utterance where each phoneme is pronounced in a speaker-independent way because its mel features are obtained by averaging mel

features corresponding to this phoneme across all speakers. This novel disentanglement method along with powerful diffusion-based decoding leads to the state-of-the-art zero-shot voice conversion algorithm.

Despite their excellent performance in terms of generation quality in many domains, diffusion models are notorious for their low inference speed [7]. Classic DPMs generate data according to iterative procedure which can be viewed as solving certain differential equation in data domain. Depending on additional requirements on computational budget and properties of the objects to be generated, it can be either a stochastic differential equation (SDE) or an ordinary differential equation (ODE) [2, 30]. When general-purpose differential equation solvers are used, relatively small step sizes are necessary to obtain generated data samples of sufficiently good quality. It means that sampling from a diffusion model requires relatively large number of iterations, typically no less than several hundreds [2, 7], meaning that there should be at least several hundreds forward passes through the trained neural network which makes inference very slow. In order to make TTS and VC diffusion models proposed in this work faster, we invent two methods:

• Replacing conventional variance preserving (VP) diffusion model [2] with the generalized mean-reverting variance preserving (MR-VP) one. Whereas in VP DPMs differential equation is solved with initial condition from the standard normal prior N(0,1), MR-VP DPM is designed in such a way that its prior is NI) where ^ is some tensor having the same dimensionality as data. Thus, if we choose MR-VP DPM as the decoder in Grad-TTS and set mean ^ of the prior to expanded encoded text sequence (i.e. rough target mel-spectrogram), then differential equation will be solved with initial condition that is already close to the desired mel-spectrogram in some sense which can potentially facilitate numerical solution of this equation. In this work we derive all necessary formulae for MR-VP DPM training and inference and show its benefits over standard VP DPM through subjective human evaluation.

• Utilizing DPM-specific maximum likelihood (ML) differential equation solver instead of general-purpose solvers like Euler or Euler-Maruyama [31]. It has been established [2, 32] that continuous-time trajectories of reverse diffusion process (the one which generates data from the prior) are equal in distribution to those of forward diffusion process (the one which gradually adds Gaussian noise to data) under some mild conditions. So, continuous-time reverse diffusion parameterized by DPM backbone trained till optimality is the model maximizing log-likelihood of continuous-time trajectories of forward diffusion. However, sampling from a diffusion model means solving a differential equation corresponding to reverse diffusion, which in its turn

leads to discrete-time Markov chain that can lose the mentioned maximum likelihood property. Loosely speaking, ML solver is defined as the solver maximizing log-likelihood of discretized forward diffusion trajectories over a wide class of solvers including Euler-Maruyama one. As seen from this informal definition, ML solver has some optimal properties with respect to the distribution of forward diffusion trajectories, so, unlike many popular solvers used in diffusion-related literature, it is no longer a general-purpose one. In this work we properly define ML solver, write down and prove some of the properties regarding its optimality and show empirically that it is better than several popular general-purpose solvers in terms of generation quality when number of solver steps is limited to a very small value.

The final part of the work focuses on the diffusion-based voice conversion model and its further improvement. It has lately been hypothesized [33] that VP diffusion trajectories obtained when generating data with VP DPM by solving ODE perform optimal transport [34] between the prior and data distribution (given that discretization error is zero and DPM backbone is trained till optimality). Although there are only empirical evidences supporting this statement and, moreover, it has been proven that this optimal transport property is only approximate and does not hold precisely for complex data distributions [35], we still find that for our MR-VP DPM underlying the VC model making an assumption about optimal transport property of its ODE trajectories can help to improve conversion quality, get rid of some mispronunciation issues and improve prosody. In this work we show how exactly the mentioned assumption can be utilized to enhance voice conversion and perform subjective human evaluation to justify the proposed method.

Goal of the work

In summary, the goal of this work is to study benefits and drawbacks of speech synthesis systems largely relying on diffusion probabilistic models, and propose new methods of overcoming main of these drawbacks.

Key results and conclusions

Contributions. The contribution of this work is mostly fourfold:

1. Two speech synthesis systems (solving the text-to-speech and voice conversion tasks) based on diffusion probabilistic models are proposed. They show results competitive with those of the contemporary state-of-the-art algorithms (for the text-to-speech task) or

better than those (for the voice conversion task) thus constituting a valuable empirical contribution for the speech synthesis community.

2. Conventional variance preserving diffusion probabilistic models are generalized to a wider class of models with the richer prior. We provide the necessary apparatus to work with this class of models and show its benefits.

3. We present a novel DPM-specific differential equation solver, derive some of its properties regarding log-likelihood maximization, and demonstrate its superior performance compared to general-purpose solvers commonly used in diffusion models.

4. We show how performance of the voice conversion model can be improved by the optimal transport assumption.

Theoretical and practical significance. The speech synthesis models proposed in this work perform comparably or better than the state-of-the-art models known previously from the literature, thus yielding practical significance. Furthermore, theoretically-grounded methods of enhancing diffusion-based speech synthesis algorithms such as mean-reverting variance preserving generalization of conventional diffusion models, or maximum likelihood DPM-specific solver serving as the sampling method for diffusion models, allow for further generalization and improvement and can be used not only in speech applications but in any systems relying on diffusion probabilistic modeling.

Methodology and research methods. All the models proposed in this work were implemented with Python programming language and PyTorch deep learning framework. Their training and testing was done according to the standard methodology of machine learning. Standard apparatus of probability theory and stochastic calculus was used to derive theoretical results. Finally, subjective human evaluation was performed to assess quality of synthesized speech, and results of this evaluation were processed by means of mathematical statistics.

Reliability of the declared results comes from rigorous proofs of theoretical statements supporting them, experimental setups common for speech-related literature, publicly available codes corresponding to some of the proposed algorithms, and the clear and unambiguous way these algorithms are presented in this work.

Key aspects/ideas to be defended:

1. The text-to-speech model Grad-TTS having DPM-based decoder and performing on par with contemporary state-of-the-art text-to-speech models.

2. The diffusion-based voice conversion model capable of zero-shot conversion outper-

forming contemporary state-of-the-art zero-shot voice conversion models.

3. The generalization of variance preserving DPMs with wider class of priors allowing for faster synthesis.

4. The maximum likelihood DPM-specific solver allowing for faster synthesis.

5. The voice conversion model enhancement relying on the optimal transport assumption.

Personal contribution. All algorithms and theoretical statements are obtained by the author. In particular, the author has formulated and proven the theorem included in this work. The design of the voice conversion model also belongs to the author. The design of the text-to-speech model as well as the experimental part including training setup, evaluation and codes is the result of the collaboration between all coauthors of the papers. Text of these papers is fully written by the author of this work.

First-tier publications:

[A] Vadim Popov*, Ivan Vovk*, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech // Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 8599-8608, 2021. CORE rank A* conference.

[B] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, Jiansheng Wei Diffusion-based Voice Conversion with Fast Maximum Likelihood Sampling Scheme / International Conference on Learning Representations (ICLR), 2022. CORE rank A* conference.

[C] Vadim Popov, Amantur Amatov, Mikhail Kudinov, Vladimir Gogoryan, Tasnima Sadekova, Ivan Vovk Optimal Transport in Diffusion Modeling for Conversion Tasks in Audio Domain / Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP49357.2023.10094854, 2023. Scopus Q1 database.

* — authors with equal contribution.

Volume and structure of the work. The thesis contains an introduction, contents of publications and a conclusion. Section 1 describes TTS model presented in paper [A] and DPM modifications related to it. Section 2 is devoted to maximum likelihood reverse SDE solver introduced in paper [B] and VC model studied there. Finally, Section 3

contains results on optimizing voice conversion procedure based on the optimal transport assumption considered in paper [C]. The full volume of the thesis is 228 pages.

Заключение диссертации по теме «Другие cпециальности», Попов Вадим Сергеевич

Заключение

Основные результаты данной работы можно сформулировать следующим образом:

1. Было предложено обобщение сохраняющей дисперсию диффузионной вероятностной модели до диффузионной модели с более богатым классом априорных распределений. Обобщённые диффузии затем легли в основу двух моделей синтеза речи, решающих задачи синтеза речи по тексту и переноса голоса по единственному речевому фрагменту. Качество синтеза предложенных моделей оказалось сравнимо с качеством лучших современных им моделей, что было продемонстрировано в целом ряде экспериментов. Кроме того, было проведено сравнение с моделью синтеза речи по тексту на основе стандартной сохраняющей дисперсию диффузионной модели, показавшее, что удачный выбор среднего априорного распределения в обобщённой диффузионной модели позволил ускорить синтез речи. Таким образом, предложенное обобщение упомянутого класса диффузионных моделей может оказаться полезным в случае, когда можно сделать предположение об оптимальных с точки зрения скорости генерации параметрах априорного распределения.

2. Был введён новый метод решения стохастических дифференциальных уравнений, максимизирующий логарифм правдоподобия дискретизованных траекторий прямой диффузии, и приведены все результаты и формулы, необходимые для его использования в наиболее популярных типах диффузионных вероятностных моделей. В частности, введённый метод был применён к модели переноса голоса и показал преимущества перед популярными общими методами решения дифференциальных уравнений с точки зрения количества шагов метода, необходимых для синтеза речи высокого качества.

3. Был изучен метод переноса голоса, основанный на предположении об оптимальном решении транспортной задачи диффузионной вероятностной моделью. Несмотря на хорошие результаты, показанные диффузионной моделью переноса голоса, некоторые проблемы, присущие ей, такие как неправильное произношение в ряде сложных случаев и неточное сохранение просодии исходного речевого фрагмента, были следствием проблем энкодера, обучаемого с помощью модели распознавания речи, и оставались неразрешёнными. Проведённая субъективная оценка восприятия речи показала, что изученный метод переноса голоса приводит к улучшению качества синтеза и частичному преодолению упомянутых проблем вследствие, вероятно, того, что он в некотором приближении осуществляет оптимальное решение транспортной

задачи между исходным и целевым голосами.

Список литературы диссертационного исследования кандидат наук Попов Вадим Сергеевич, 2026 год

Список литературы

[1] Sohl-Dickstein J., Weiss E.A., Maheswaranathan N. et al. Deep Unsupervised Learning using Nonequilibrium Thermodynamics / Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 2256-2265, 2015.

[2] Song Y., Sohl-Dickstein J., Kingma D.P et al. Score-based Generative Modeling through Stochastic Differential Equations / International Conference on Learning Representations (ICLR), 2021.

[3] Liptser R.S. and Shiryaev A.N. Statistics of Random Processes // Stochastic Modelling and Applied Probability, volume 5. Springer-Verlag, 1978.

[4] Hyvarinen A. Estimation of Non-Normalized Statistical Models by Score Matching / Journal of Machine Learning Research, vol. 6, pp. 695-709, 2005.

[5] Goodfellow I., Bengio Y. and Courville A. Deep Learning // MIT Press, 2016.

[6] Dhariwal P. and Nichol A.Q. Diffusion Models Beat GANs on Image Synthesis / Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 8780-8794, 2021.

[7] Ho J., Jain A. and Abbeel P. Denoising Diffusion Probabilistic Models // Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 6840-6851, 2020.

[8] Ho J., Salimans T., Gritsenko A. et al. Video Diffusion Models / Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 8633-8646, 2022.

[9] Vignac C., Krawczuk I., Siraudin A. et al. DiGress: Discrete Denoising Diffusion for Graph Generation / International Conference on Learning Representations (ICLR), 2023.

[10] Hoogeboom E., Satorras V.G., Vignac C. et al. Equivariant Diffusion for Molecule Generation in 3D / Proceedings of the 39th International Conference on Machine Learning (ICML), pp. 8867-8887, 2022.

[11] Das A., Yang Y., Hospedales T. et al. ChiroDiff: Modelling Chirographic Data with Diffusion Models / International Conference on Learning Representations (ICLR), 2023.

[12] Hunt A.J. and Black A.W. Unit Selection in a Concatenative Speech Synthesis System Using a Large Speech Database // Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), vol. 1, pp. 373-376, 1996.

[13] Zen H., Tokuda K. and Black A.W. Statistical Parametric Speech Synthesis // Speech Communication, vol. 51(11), pp. 1039-1064, 2009.

[14] Rabiner L.R. and Schafer R.W. Introduction to Digital Speech Processing / Foundations and Trends in Signal Processing, vol. 1, pp. 1-194, 2007.

[15] Davis S.B. and Mermelstein P. Comparison of Parametric Representation for Monosyllabic Word Recognition in Continuously Spoken Sentences // IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 28(4), pp. 357-366, 1980.

[16] Makhoul J. Linear Prediction: A Tutorial Review // Proceedings of the IEEE, vol. 63(4), pp. 561-580, 1975.

[17] Dudley H. Remaking Speech // The Journal of the Acoustical Society of America, vol. 1(2), pp. 169-177, 1939.

[18] Ling Z.-H., Kang S.-Y., Zen H. et al. Deep Learning for Acoustic Modeling in Parametric Speech Generation: A Systematic Review of Existing Techniques and Future Trends // IEEE Signal Processing Magazine, vol. 32(3), pp. 35-52, 2015.

[19] Tokuda K., Nankaku Y., Toda T. et al. Speech Synthesis Based on Hidden Markov Models // Proceedings of the IEEE, vol. 101, pp. 1234-1252, 2013.

[20] Oord A., Dieleman S., Zen H. et al. WaveNet: A Generative Model for Raw Audio / 9th ISCA Speech Synthesis Workshop, 2016.

[21] Shen J., Pang R., Weiss R.J. et al. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions // Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783, 2018.

[22] Kim J., Kim S., Kong J. et al. Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search / Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 8067-8077, 2020.

[23] Ren Y., Ruan Y., Tan X. et al. FastSpeech: Fast, Robust and Controllable Text to Speech / Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 3171-3180, 2019.

[24] Luong T., Pham H. and Manning C.D. Effective Approaches to Attention-based Neural Machine Translation / Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1412-1421, 2015.

[25] Kingma D.P. and Welling M. Auto-Encoding Variational Bayes // International Conference on Learning Representations (ICLR), 2014.

[26] Rezende D.J. and Mohamed S. Variational Inference with Normalizing Flows / Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 1530-1538, 2015.

[27] Goodfellow I., Pouget-Abadie J., Mirza M. et al. Generative Adversarial Nets / Proceedings of the 28th International Conference on Neural Information Processing Systems, pp. 2672-2680, 2014.

[28] Chen Y.-H, Wu D.-Y, Wu T.-H. et al. Again-VC: A One-Shot Voice Conversion Using Activation Guidance and Adaptive Instance Normalization / Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5954-5958, 2021.

[29] Lin Y.Y., Chien C.-M., Lin J.-H. et al. FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments with Attention / Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5939-5943, 2021.

[30] Song J., Meng C. and Ermon S. Denoising Diffusion Implicit Models / International Conference on Learning Representations (ICLR), 2021.

[31] Kloeden P.E. and Platen E. Numerical Solution of Stochastic Differential Equations / Stochastic Modelling and Applied Probability, volume 23. Springer-Verlag, 1992.

[32] Anderson, B.D.O. Reverse-time Diffusion Equation Models / Stochastic Processes and Their Applications, vol. 12(3), pp. 313-326, 1982.

[33] Khrulkov V., Ryzhakov G., Chertkov A. et al. Understanding DDPM Latent Codes through Optimal Transport / International Conference on Learning Representations (ICLR), 2023.

[34] Villani C. Optimal Transport: Old and New / Springer, 2016.

[35] Lavenant H. and Santambrogio F. The Flow Map of the Fokker-Planck Equation does not Provide Optimal Transport / Applied Mathematics Letters, volume 133, 2022.

[36] Oord A., Li Y., Babuschkin I. et al. Parallel WaveNet: Fast High-Fidelity Speech Synthesis / Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 3918-3926, 2018.

[37] Prenger R., Valle R. and Catanzaro B. Waveglow: A Flow-based Generative Network for Speech Synthesis / Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621, 2019.

[38] Yamamoto R., Song E. and Kim J.-M. Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram / Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199-6203, 2020.

[39] Kumar K., Kumar R., de Boissiere T. et al. MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis / Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 14910-14921, 2019.

[40] Kong J., Kim J. and Bae J. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis / Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 17022-17033, 2020.

[41] Li N., Liu S., Liu Y. et al. Neural Speech Synthesis with Transformer Network / Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6706-6713, 2019.

[42] Elias I., Zen H., Shen J. et al. Parallel Tacotron: Non-Autoregressive and Controllable TTS / Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5709-5713, 2021.

[43] Shen J., Jia Y., Chrzanowski M. et al. Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling / ArXiv, abs/2010.04301, 2020.

[44] Rabiner L.R. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition / Proceedings of the IEEE, vol. 77(2), pp. 257-286, 1989.

[45] Kingma D.P. and Dhariwal P. Glow: Generative Flow with Invertible 1x1 Convolutions / Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 10236-10245, 2018.

[46] Chen N., Zhang Y., Zen H. et al. WaveGrad: Estimating Gradients for Waveform Generation / International Conference on Learning Representations (ICLR), 2021.

[47] Kong Z., Ping W., Huang J. et al. DiffWave: A Versatile Diffusion Model for Audio Synthesis / International Conference on Learning Representations (ICLR), 2021.

[48] Song Y. and Ermon S. Generative Modeling by Estimating Gradients of the Data Distribution / Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 11918-11930, 2019.

[49] Ronneberger O., Fischer P. and Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation / Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234-241, 2015.

[50] Ito K. The LJ Speech Dataset, 2017.

[51] Binkowski M., Donahue J., Dieleman S. et al. High Fidelity Speech Synthesis with Adversarial Networks / International Conference on Learning Representations (ICLR), 2020.

[52] Chen R.T.Q., Rubanova Y., Bettencourt J. et al. Neural Ordinary Differential Equations / Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6571-6583, 2018.

[53] Qian K., Zhang Y., Chang S. et al. AutoVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss / Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 5210-5219, 2019.

[54] Qian K., Jin Z., Hasegawa-Johnson M. et al. F0-Consistent Many-To-Many NonParallel Voice Conversion via Conditional Autoencoder / Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6284-6288, 2020.

[55] Nercessian S. Improved Zero-Shot Voice Conversion Using Explicit Conditioning Signals / Proceedings of Interspeech, pp. 4711-4715, 2020.

[56] Chou J.-C. and Lee H.-Y. One-Shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization / Proceedings of Interspeech, pp. 664-668, 2019.

[57] Ishihara T. and Saito D. Attention-based Speaker Embeddings for One-Shot Voice Conversion / Proceedings of Interspeech, pp. 806-810, 2020.

[58] Liu S., Cao Y., Wang D. et al. Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling / IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1717-1728, 2021.

[59] Wu D.-Y., Chen Y.-H. and Lee H.-Y. VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net Architecture // Proceedings of Interspeech, pp. 4691-4695, 2020.

[60] Wang D., Deng L., Yeung Y.T. et al. VQMIVC: Vector Quantization and Mutual Information-based Unsupervised Speech Representation Disentanglement for One-Shot Voice Conversion // Proceedings of Interspeech, pp. 1344-1348, 2021.

[61] Luong M. and Tran V.A. Many-to-Many Voice Conversion Based Feature Disentanglement Using Variational Autoencoder / Proceedings of Interspeech, pp. 851-855, 2021.

[62] Saito Y., Ijima Y., Nishida K. et al. Non-Parallel Voice Conversion Using Variational Autoencoders Conditioned by Phonetic Posteriorgrams and D-Vectors // Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5274-5278, 2018.

[63] Nichol A.Q. and Dhariwal P. Improved Denoising Diffusion Probabilistic Models / Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 8162-8171, 2021.

[64] Lee S.-G., Kim H., Shin C. et al. PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior / International Conference on Learning Representations (ICLR), 2021.

[65] McAuliffe M., Socolof M., Mihuc S. et al. Montreal Forced Aligner: Trainable TextSpeech Alignment Using Kaldi // Proceedings of Interspeech, pp. 498-502, 2017.

[66] Zen H., Clark R., Weiss R.J. et al. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech / Proceedings of Interspeech, pp. 1526-1530, 2019.

[67] Jia Y., Zhang Y., Weiss R.J. et al. Transfer Learning from Speaker Verification to Multispeaker Text-to-Speech Synthesis // Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 4485-4495, 2018.

[68] Song Y., Durkan C., Murray I. et al. Maximum Likelihood Training of Score-based Diffusion Models / Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 1415-1428, 2021.

[69] Henry-Labordère P., Tan X. and Touzi N. Unbiased Simulation of Stochastic Differential Equations // The Annals of Applied Probability, vol. 27, pp. 3305-3341, 2017.

[70] Beskos A. and Roberts G.O. Exact Simulation of Diffusions // The Annals of Applied Probability, vol. 15, pp. 2422-2444, 2005.

[71] Kingma D.P., Salimans T., Poole B. et al. Variational Diffusion Models // Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 21696-21707, 2021.

[72] Yamagishi J., Veaux C. and MacDonald K. CSTR VCTK Corpus: English Multispeaker Corpus for CSTR Voice Cloning Toolkit (version 0.92), 2019.

[73] Baevski A., Zhou Y., Mohamed A. et al. Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations / Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 12449-12460, 2020.

[74] Kim K.-W., Park S.-W. and Joe M.-C. Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniques // Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6997-7001, 2022.

[75] Panayotov V., Chen G., Povey D. et al. Librispeech: An ASR Corpus Based on Public Domain Audio Books / Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015.

[76] Su X., Song J., Meng C. et al. Dual Diffusion Implicit Bridges for Image-to-Image Translation / International Conference on Learning Representations (ICLR), 2023.

[77] Lu C, Zhou Y., Bao F. et al. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps / Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 5775-5787, 2022.

[78] Bitton A., Esling P. and Chemla-Romeu-Santos A. Modulated Variational AutoEncoders for many-to-many musical timbre transfer / ArXiv, abs/1810.00222, 2018.

[79] Chang Y.-C., Chen W.-C. and Hu M.-C. Semi-supervised Many-to-many Music Timbre Transfer / Proceedings of the 21st International Conference on Multimedia Retrieval (ICMR), pp. 442-446, 2021.

[80] Huang S, Li Q, Anil C. et al. TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer / International Conference on Learning Representations (ICLR), 2019.