ML for Creative Writing, Cont.

“You shall know a word by the company it keeps!” is the historically common approach to linguistic modeling. It uses word vectoring, or word embedding, to train a set of word vectors based on a large corpus of text. With each word represented by a point in the embedding space, these points are learned and moved around based on the words surrounding the target word. Defining a word by the company it keeps allows word embedding to learn something about the meaning of words. The vector space representation of the words provides a projection where words with similar meanings are locally clustered within the space.

Whereas recurrent neural network (RNN) models can be trained from scratch, most recent models are pre-trained and then transferred to downstream tasks. Rather than word vectoring, pre-training is done entirely on data-rich tasks to develop general-purpose abilities and knowledge. It can be done via supervised learning on a large labeled dataset, but modern techniques using unsupervised learning on unlabeled data is most attractive because of the massive amount of text data available on the web.

Text-to-text Transfer Transformer (T5) is a text-to-text transformer model, producing a block of output text given a block of input text, in contrast to BERT-style models that can only output either a class label or a span of the input. T5 is pre-trained on a wide set of tasks, including language translation, linguistic acceptability, sentence similarity, documentation summarization, question answering, fill-in-the-blank, and sentiment labeling. It is trained on the Colossal Clean Crawled Corpus (C4) dataset, a cleaned version of Common Crawl that is twice larger than Wikipedia.

While training the RNN model required a graphical processing unit (GPU), to fine-tune a T5 model requires multiple tensor processing units (TPU), which are network-connected accelerators with 8 compute cores. Utilizing default parameters, the pre-trained model can be plug-and-play in data science speak, but modifying parameters upon the dataset also influences model training behaviors and predictions.

Using the Hans Christian Andersen’s fairytales to train the model to predict the next span of text given a preceding span of text, the dataset parameters include the length of input and output words and the step sizes between each input-output pair:
   + The input step size determines the word count between each input-output pair. If the input step size equals the input length, each input block of text is unique and there is no overlapping between each input.
    + The prediction step size determines the overlapping count between input and output. If the prediction step size equals the input length, the output is the next block of text following the input and there is no overlapping.
   + The input and output lengths determine the size of the tokenized dataset. T5 allows for a maximum sequence length of 512 words.
   + All together, input and prediction step size and input and output sequence lengths determine the total length of the dataset, which affects the amount of time and resources required to train the model upon each epoch.

I trained the model on a range of sequence length, from 1 to 512. I also modified the input step based on the sequence length, because it affects the dataset length. For example, a dataset with sequence length of 512 words has 545 unique pairs, but with input step of 2, its total length is 126820. A dataset with sequence length of 128 words has 2713 unique pairs, but with input step of 3, its total length is 113141. The goal was to keep the total length between 100000-150000 because it appeared to be the most efficient to train the model for each epoch. It also means that the model is trained differently for each sequence length. It’s repeated 247 times for each epoch with sequence length of 512, but 41 times with 128. This combined with the difference in sequence lengths make it more difficult to compare accuracy evaluation among the datasets. A larger repetition during training leads to better accuracy, but on the contrary, greater sequence lengths produce lower accuracy during evaluation because there is more text to generate. Since this experiment is about creative writing, thus prediction accuracy is not of priority, we can look at training loss values to compare the datasets.

T5 uses the cross-entropy loss functions to compute training loss. From the concept of information theory, entropy is the number of bits required to transmit a randomly selected event from a probability distribution. A skewed distribution has a low entropy, whereas one whose events of equal probability has a larger entropy. Cross-entropy computes the number of bits required to represent or transmit an event from one distribution in comparison to another distribution. Cross-entropy loss measures the performance of a model whose output is a probability value between 0 and 1. The loss value increases as the predicted probability diverges from the actual prediction. A perfect model would have a loss value of 0.

Previously, the RNN model was trained from scratch to predict characters with sequence length of 100 characters. It performed fair after 30 epochs of training, achieving loss values toward .6. On the other hand, the fine-tuned T5 model broke down with very small sequence lengths of 1 and 16 words, having loss values hovered between 3.5 to 2. It performed much better with larger sequence lengths. Between sequence lengths of 128 to 512 words, training loss values decreased toward .1 after 10 epochs.

Here are the outputs for sequence lengths of 128, 256, and 512 after 10 epochs of training, generated for 5 iterations. I’ve taken the liberty to rearrange the sentences for coherence.

Input Text:
In the garden all the apple-trees were in blossom. They had hastened to bring forth flowers before they got green leaves, and in the yard all the ducklings walked up and down, and the cat too: it basked in the sun and licked the sunshine from its own paws. And when one looked at the fields, how beautifully the corn stood and how green it shone, without comparison! and there was a twittering and a fluttering of all the little birds, as if the day were a great festival.

Output Text:

seq length = 128; training input step = 3; prediction step = 128
Subject: The pigeons’ prayer
Theme: God, prayer, heaven, hunger, happiness

The sun was so bright and warm, and the air so mild and fresh. The pigeons sat and cried, “Pray, pray,” and all the birds sang the same tune.
“I am very thankful for all that God is doing for me, and that I may never forget my good fortune. I have been so wonderfully blessed. I do not suffer from hunger. I have even seen the roses in the garden.”
The sun shone upon the birds, the clouds hung low, and the pigeons sat down on the grass. “I am well provided for. Everything is good for me. I am the happiest of all creatures.”

seq length = 256; training input step = 3; prediction step = 256
Subject: Jurgen, the fisherman
Theme: illness, misery, and the cure

Jurgen was a lark, and had plenty of courage, and a good swimmer, too. He was a fisherman and made his family proud; but he was ashamed of himself. One day, said Jurgen, “I have had enough. They are proud of me, but that does not matter. I have had enough of myself.” So he left for the sand-hills, where he treaded on fluff for hours, and then toward the heath where the storks lived. “The poor fisherman is miserable,” said the little stork, “for he can do nothing but run away.” The stork sang in a sweet voice, that Jurgen might be a better man than this, that he was greatly ill and had to walk about the lotus-flower which grew in the wood. After that he felt quite well again and was allowed to go on at once when the sun was high enough for him to penetrate the thick sand. The birds sang in swarms, “We are going to cause some storms to come. Go away. Nobody knows us, and we know that we are well off.”

seq length = 512; training input step = 2; prediction step = 512
Subject: Clara, the mouse
Theme: beautiful things and sameness, loneliness, Sunday and wedding

A little sprig or a little ruff; the ducklings sprang from the tree and stood among the green leaves. The ducks would have had something else to do than to sit upon a stick, and to look at the flowers as if they did not like to tread upon them. In the garden everything was bright and beautiful with blue and red flowers, corn and sunflowers. “This is all very beautiful. The flowers are always the same, and the air is always the same. I cannot endure it,” said Clara, “it is so comfortable here.” It is sometimes very lonely in the house where the sun shines and the grass grows.

In the kitchen there were a number of beautiful things, both great and small, and each piece of furniture had his own ornaments. Clara seated herself on the edge of the chair and dreamed of the golden wedding.

I went out into the garden with my little friend. All night she was in the garden. When the wind blew and the air became clear, Clara was so delighted; she felt quite happy at the thought that she was going to be married. She contemplated the beautiful flowers and thought of the perfect day.

One day – it was Sunday – the golden wedding day. I could see the green leaves and the bells ringing as if they were singing songs. Little Miss Clara sat by her side, and her father kissed her lips. Her mother stood by the window looking at the bright blue flowers and wished Clara all the happiness. Then the mice went out to the green wood, and listened to the blue and the red flowers, and wished her farewell.

——————————————————————-

Original Output Text:

seq length = 128; training input step = 3; prediction step = 128
Run time: 11.974763870239258
the sun was shining, and the pigeons sat and cried “Pray, pray,” and all the birds sang the same tune; and the little birds too. “I am very thankful that God is doing all that for me, and that I may live for ever, and never forget my good fortune.” “Yes, indeed,” said the little boy in the house; “I have been so wonderfully blessed; I have even seen the roses in the garden, and have been so blessed by my kind soul. I cannot imagine what happiness is to come for me, but I know that I must the pigeons, and the little birds too. “It is wonderful what you have done, for it is so God-like.” And then the little bird said, “Pray, pray.” And the sun was so bright and warm, and the air so mild and fresh, that it made one’s eyes ache. “It is wonderful what you have done,” said the little boy. “I have been so blessed indeed. I am not able to live in poverty, but to have the strength to do so. I am well provided for. Everything is good for me, and I a little pigeon. “I am the happiest of all creatures,” said the little bird, “and I am going to heaven.” “I am the happiest of all creatures,” said the little bird, “and all the good that God has created. I am going to heaven.” And the little bird prayed also, and nodded his head as he did so. “I am the happiest of all creatures,” said the little bird, “and I am always glad when I see the bright sunshine.” “I am the happiest of men,” said the pigeon sat down, and listened attentively. “I am not hungry, I am too fat,” said the little bird, “I cannot do otherwise.” “I am going to heaven, and I shall be able to do so, although poverty and oppression keep me from my advantage. I am going to heaven, and all the good that God has created.” “Ah, yes; everything is good for me,” said the little pigeon, “and I am going to heaven.” “I am going to heaven, and all the good that God has created.” ” he was frightened, and screamed, and cried, “Here I am. I am not in the least unhappy. I cannot do otherwise.” “It is very unfortunate,” said the pigeon. “I cannot do otherwise.” And then he flew away. The sun shone upon the birds, the clouds hung low, and the pigeon sat down on the grass. “I am happy,” said the little bird, “I am not afraid. However, I will not suffer hunger or hunger. I cannot do otherwise.” “Go

seq length = 256; training input step = 3; prediction step = 256
Run time: 24.323667287826538
a little lark. He was a lark, and had plenty of courage, and might be a good swimmer, too, for he was strong enough. And the ducklings ran out into the yard and all the birds sat upon it and played with it, and the cat sat by itself, and nodded in the sunshine. “I am no judge of myself,” said the little cat. “I have had enough of it, and I have had enough. It is true that I did not lie out here in the cold, and I am ashamed of myself. I wish I had a little dog that was not afraid. He was strong enough, and had plenty of strength, and could do no harm to any one, except the ducks and the cat. And the ducklings twittered and chirped, and they cried, “Tweet, tweet;” but that did not matter. In the same hour there came a sound of crying and cries of anguish and scream; it was like the sound of an organ, or the scream of an organ, or the shrillness of an organ, or the a little swan, and he was a good swimmer. Then the little cat hopped out and played with the swan and the cat, and all the ducks and the cat ran and floated away, and nodded in the sunshine. “I have had enough of it,” said the little stork, “and I am ashamed of it. It cannot be said of a dog, unless he is a swan. He belongs to the family and is named Jurgen. It is true that he was a very little fellow, and therefore I have not had the honor of becoming a member of the family. I have had enough of it, and I am proud of it.” “It is true that the ducks are not afraid of me,” said Jurgen; “I have had enough. They are proud of me, but that does not matter. I have had enough of myself.” Then Jurgen went to the sand-dunes and went to the heath where the storks lived; there were so many storks and storks that could do nothing. storks could make no sense out of this the stork, “the poor fisherman is miserable.” “He must be miserable,” said the little stork, “for he can do nothing but run away and be trampled upon. He should be rewarded. I mean the poor fisherman; if I am not able to do so, I will go and fetch money.” And the poor fisherman was frightened at this, and ran away and played with the little cat, and ran and floated away. “You poor creature,” said the little stork, “you have done more harm than good. You belong to the family and are named Jurgen; you belong to the stork family; you belong to the stork family, and you belong to the stork family; you belong to the stork family, and therefore I have not been able to stay here for a whole week. It is unfortunate for you that I have been allowed to stay here for one whole night, and that I am too, although I am well off with all that. Yet I have lived in the stork family for many years, and I have enjoyed myself, but no longer to be ill he was a very good man, they did not deserve to be treated like such a little person. “The young ones are very good, although they are often scratching and kicking each other, which is no one’s business but a pleasure.” And then the sun shone and the stork sang in a sweet voice, that Jurgen might be a better man than this. He went and got up there, and went and flew away to the sand-hills, where he had been treading on by the fluff. The storks were angry at his exploits, and said to one of them, “You are doing too much for yourself; you deserve to be rewarded, and that is why I have been so miserable.” “But why have you done more harm than good?” said the stork. “I mean nothing at all,” said the cat; “I mean nothing at all. It is horrible, but I must try and help myself. I mean everything; I have done nothing at all, I mean nothing at all. You are doing nothing at all, poor me.” And then the storks punished him, and said a great he was ill, and had to walk about the lotus-flower which grows in the wood, for he and his foster-mother had been there, and had not been ill. “Well, now, it is getting better and better,” said the stork-mamma; and after that he felt quite well again, and was allowed to go on for one whole night, and therefore did not suffer. But he was allowed to go on at once, and the sun was high enough for him to penetrate the thick sand, and the birds sang in swarms, “We are going to cause some storms to come,” and one day two little birds joined their hands and screamed, “Go away. Nobody knows us, and we know that we are well off.” Then stork-papa said to the little birds, “If we had not been allowed to go so far, it would have been much better.” “Well, now we have to go farther than you have;” and he went on. Then he flew to the sand-hills, where he had treading on fluff for hours, while

seq length = 512; training input step = 2; prediction step = 512
Run time: 48.81150579452515
a little sprig or a little ruff; the ducklings sprang from the tree and stood among the green leaves. “Here are my experiences,” said the cat; “they are not always happy. I know that you are always right. One day I went out into the sunshine with my little friend who was going to visit me at her house. It was Sunday. My friend said that the little chickens would have liked to have green leaves, but the apple-trees would have shone in green sunshine while I was with my mother; she had also been a lark, as I had a turkey-cock, but she said nothing. The other children were very much pleased with this flower, and they both worked for my garden. But the ducks would have had something else to do than to sit upon a stick, and to look at the other flowers as if they did not like to tread upon them. But the ducks could not get so far, they looked at the sun in their own way. It was quite a pleasure to be with them; they were as happy as we are with them. There was a grand lady behind the door, who looked as if she had just come out of the house. She had her own room, and felt so comfortable that she did not feel at home. How charming it was! All the little mouse-cocks shook their heads, and cried, “Tweet, tweet!” And the little mouse was very much pleased. “I cannot endure it,” said the mouse, “it is so comfortable here, and yet it is so comfortable.” And the mouse looked at the sun, and smiled at it, but there was a difference. There was a difference also, for every little creature had green leaves, and in the peacock’s feathers, and a kiss from the other–the corn and the sunflowers–were they had both been happy. The turkeys both flew to the east, and ate all their hearts; the turkeys looked the same way, the same bells flitted to and fro on their hind legs; there was a difference also, for the turkey-cock was scrambling at the top. “I can’t help laughing,” said the turkey; “I can’t, and it’s all my faults.” a little ruff, or a little ruff, that is to say, good-bye to the world. I have been to the duckyard, to see my mother’s little granddaughter, who is quite old; she has been here many years, and she has read all the newspapers. She is always at home, and pleasant to her, but she does not know that I am always at home. It is sometimes very lonely in the house where the sun shines and the grass grows. I wish I could stay at home, but it is sometimes very lonely. I wish I were not lying in the cold, standing in the cold, looking at the sun, and looking at the green leaves. Then I went to the hay-tree and planted some of it. It seemed to me as if the whole world were to be lighted up, and to see all the little mice, who could relate stories as they did to the little mice. “You are quite right,” said the stork, “but you are not alone, and all the same. You are only for one day, when everything is bright and new, and you are always in the same place.” “Yes, you may be very happy,” said the little mouse, “but you are not like other mice. You are only like other mice. I know each other very well; they do not know that I am the one who has told me so. One day–it is Sunday–I went out into the garden with my little friend. She was going to her wedding. It was Sunday, and she was going to be married, so that she might be married. But the other flowers, who were also called, looked very beautiful, and looked like flowers. Then she stood looking at the sun, and nodded to the little mouse, “Creak, creak,” as she had already been told by the other mice. “Many thanks for the merry but very little mice,” said the other. “Now let us go straight to the other flowers.” And so it went with them; but the little mouse was not at all uneasy. “We cannot endure it,” said the other mouse, “we must give up our little comforts. The little mouse is very much like a mother. She does not like to be looked at, and to know that she has been properly dressed. She has had a taste like that in the storks, “it is always the same in the yard,” said he. “But it is always the same place. I don’t know him, and I cannot laugh. He is always in the same place. He has flown to the other side, and sat upon the floor, and listened to all the conversation. It seemed to me as if I could be one of the family, and wished himself one of the family. But in the garden everything was bright and beautiful. I could see the green leaves, and the bells ringing from the other side, as if they were singing songs. Then the mouse went out to the green wood, and listened to the blue and the red flowers, and wished himself farewell. Little Miss Clara sat by her side, and her father kissed her lips; she was so delighted, that she fell in love with the little mice, and wanted to marry the princess; but she was not at all interested in this world, and was so delighted that she got to go into the garden with her mother. “There is something so beautiful between us,” said she, “and very beautiful.” “Yes, that is very beautiful,” said the little mouse, “but sometimes it is too much for a mouse to be made to laugh. It is easy to feel that if you are not always laughing, it will make you feel as if you were always going to be married. I have often listened to all the news, and always to hear the parlor-mice, and sometimes it is quite as pleasant as sitting on the floor as I do with the other mice. I can relate some of it, but it is not my fault that I am wrong. Sometimes I am wrong, but it is not my fault. I cannot, and it is all very wrong. I have given everything I have done to myself, and to be able to help myself. I hope all the others will be happy; but I cannot.” “Is that the little mouse that I am?” said the queen. “Yes, to-day is the golden wedding-day,” replied the stork-mamma; “but he died yesterday,” said he, “because he had been lying all night in the dead. All the other mice, who are only old, could be quite alone, and could not stay away. Some of them the storks. “I cannot endure it,” said the little mice, and then they took off their hats, and went straight to the flowers, which were bound together in a knot, and shook their heads. Then stork-mamma was very busy at the door, when she wanted to go with them; so she hopped out into the garden, and stood by the window looking at the bright blue flowers, and wished Clara all the happiness of all. Clara was so delighted that she felt quite happy at the thought that she had a little comfort; and so she sat down to rest on the floor, and nodded to the other little mice, and then cried, “You poor child,” and then he said, “You must not be so unhappy. This is all very beautiful. The flowers are always the same, and the air is always the same.” “You must not look like one,” said the little mice; “I think that you are looking at me.” “Why, that is very beautiful,” said the mother. “But one cannot live upon something like a garden, and the other mice cannot understand it. They are always looking for the same things, and the same quality is there.” “Look, you are always looking at one,” said she, “and there is always the same bells, no doubt,” and she laughed. “The bells are ringing all the time,” replied the stork-papa; “but you must not look at them.” “It is very beautiful to look at them,” said the little mice; “and sometimes it is as beautiful as to be seen by others.” “Perhaps it is the same,” said the stork-mamma; “but I am not going to tell you all about it.” “Yes, and I will tell you all about it,” said he, “but it is not very straightforward.” “It is very straightforward,” said Clara, and then he went away with Clara. But the prince was not at all pleased; he was so much afraid of such a little creature that he had never been in a room. In the kitchen there were a number of beautiful things, both great and small. Each of these pieces of furniture stood in each other’s way, and each of them had his own ornaments, like golden bands the storks. Clara was so delighted, that she felt herself lifted up, and ready to dance once more. She was just as excited as any of the others; but she could not help herself, for she stood on the floor, looking at the bright blue flowers, and nodding to the beautiful flowers, but she was not at all uneasy. Then she seated herself on the edge of the chair, and contemplated the beautiful flowers, and thought she must have the perfect day to live. Then she seated herself on the edge of the chair, and dreamed of the golden wedding. Clara was so startled and alarmed at the sight; but the little mice said they should not remain at home, and could not bear to think of it. All night she was in the garden, when the wind blew and the air became clear, and she was seated on the floor, in the form of a frog. The frogs immediately sprang out of the bed, and opened the window. The frogs opened, and looked into the same air, and thought that the bells were ringing all around her; and the air was filled with the frog’s bright eyes, and wished the mice all the same. All the other little mice were very much pleased, and the first blush blushed, and the second blush glimmering red flowers. “It is beautiful to me,” said the stork-mamma, “that I may live forever; there is only one or two of them; I cannot bear it, and it is all the same.” “It is very beautiful,” said the stork-mamma; “but the bright sun makes all the trees beautiful, and it makes all the same. I cannot bear it, and yet it is so miserable to be with you. You must not be unhappy at all.” “The
flowers are always in the same place,” said they; and then they threw off their hats, and went into the courtyard, where the most beautiful and merry children stood round the room. All the furniture in the room were of the same color; and Clara looked at the bright blue flowers, and thought that the bells of every one were ringing, she was not at all unhappy. When she arrived at the hall, a lady