Machine Learning for Creative Writing

I think that most of us like certainty. We prefer to know what we will do today, tomorrow, and the days after. We like novelty too, but only when we are mentally prepared for it, which is a form of certainty in itself. Uncertainty begets anxiety, a state of mind so disdained that it is interpreted as a survival mechanism resonant of days when men lived in caves and were eaten by beasts.

But the truth is that whether we like it or not, uncertainty surrounds us. A simple system is deterministic. But the more complex it becomes, the more it behaves like an indeterministic system. The world is complex. Humans are complex actors upon our world.

Those who lean into uncertainty are rewarded a novel point of view of nature. The field of statistics is one of them. Instead of the top-down approach, it extracts patterns from data to infer about the behaviors of the system. In doing so, it relies on uncertainty principles to measure how well it makes inferences. By using statistical models to translate languages, Statistical Machine Translation (SMT) outperforms Rule-based Machine Translation (RBMT), which utilizes the top-down approach. Built upon SMT, Neural Machine Translation (NMT) uses neural network models to learn a statistical model for machine translation.

Computing power has improved by the miles recently, ushering in a new era for Natural Language Processing (NLP), using NMT to translate text, to summarize a body of text, and even to write new text. And writing new text is what I should like to discuss, because it sounds to me like creative writing. I’d like to see if Machine Learning can write for me a short story, like a fairytale.

At the core, the approach is to train the model to predict the next body of text given a body of text. For example, given the sentence “Hello, world!”, if the input is “Hello,”, then the output will be “world!”. However, the spirit of uncertainty dictates that the model won’t always give the same output, hence creative writing. I’ve got a large volume of Hans Christian Andersen’s fairytales, and I’m hoping that the neural network model can be trained to write one in his style. And I have a plan in place:

1. A novel approach using the Recurrent Neural Network (RNN) model to train text prediction based on characters, rather than words. For example, given the sentence “Hello, world!”, if the input is “Hell”, then the output will be “o, world!”.

2. The state-of-the-art transformer model such as T5 is employed. Transformer models leverage transfer learning, which utilizes a pre-trained model, and then fine-tuning is built upon it.

3. Attempt to improve upon the state-of-the-art model through data categorization. Instead of training on the entire story, train the model separately to generate each major event of a story. The structure to build a story includes 6 major events:

– Exposition establishes characters and setting.

– Inciting Incident is an event putting the characters into a challenging situation.

– Rising Action or Progressive Complications is the largest part of the story, where most of the conflict takes place.

– Dilemma is the moment when a character is put into a situation where they must make an impossible choice

– Climax is the big moment! The character’s choice from the dilemma drives the outcome of the conflict.

– Denouement is the end of the story. It includes the resolution and conclusion.

My idea for the first approach comes from a tutorial on tensorflow for text generation with RNN. Given a sequence of characters, the RNN model is trained to predict the next character in the sequence. Calling the model repeatedly produces longer sequences of text. The author uses Shakespeare’s writing for their dataset. Unlike the 100000 character sample from Shakespeare’s writing, there are 122 separate stories within the book of Andersen’s fairytales, meaning I need to make sure that inputs and predictions do not bleed from one story to the next. How much of a difference does it make? -maybe a lot, maybe not; but due diligence is what I’m striving for here.

This RNN model is a sequential model where each layer is stacked linearly and one input is mapped to one output between each layer. Beyond the sequential model, graphs of layers are built with functional API to handle models with non-linear topology, shared layers, and multiple inputs or outputs. The model simply consists of 3 layers, an embedding layer to map characters onto the vector space, a GRU layer to process the embeddings into a new sequence, and a dense layer to produce the outputs.

To evaluate the model’s performance, loss and accuracy values are typically utilized. Categorical cross-entropy is used when the model predicts a discrete value, which is appropriate for text generation. The cross-entropy loss function computes the expected surprise of the observer’s subjective probabilities q upon seeing data that was generated according to true probabilities p. The objective is to maximize the likelihood of observed data and minimize the amount of surprise. Accuracy values are the percentage of correct predictions among all predictions.

If prediction accuracy is what we want, then the data is generally divided into a training, a testing, and a validation set. Since my purpose here is creative writing, the entire dataset is used for training. The model was trained for 30 epochs, each time with the full dataset. Observe that the accuracy percentage plateaus at ~75%.

I had the model produce ~10000 characters, the size for a typical short story. It is included below. Because the model was trained on characters and not words, you will see that certain words are completely made up. Shakespeare was famous for inventing new vocabularies, but for Andersen’s fairytales which were translated from Danish to English, it makes for an entertaining read. Here are some of my favorite sentences:
– for no poor pendulum of sickness, where it travels with him little way.
– I long for me to try God.
– Farewell, from the rose, and sang, and the sparrow fancies that he could hardly know more but when he could see the bride, not so to bear a little walk.
– “Oh, if I am in delicate!”
“You may be very useful.”
– After a while the water sparkled in her long love, where she listened.
Water is there!

**Start your data science project for free on Kaggle or Google Colab.

Run time: 21.22520351409912 seconds
_____________________________________________________

There lay on the mountain till he had left him in his arms to correct out in the wood; the shadow came at the old woman? Every flash of spirits sang, and pulled down their shoulders. It was Something’s pretty shoes, he sat on the water to weep and die from the top of the princess’s eyes of the Tyrmay, and had the best flower to flower, and had been a stranger; but that was one who had danced and found, and yet he seemed during the Bedlied ham. “They jumped upon you, and the candies becomes estable; for no poor pendulum of sickness, where it travels with him little way.”

“You will forget my father, that I may have done,” said she, “for then I think it would be sure to be always all day. I will grow up my carved mountains and leaves–there are two freshness, sinsing in the walls from at the door, just feel that of your misfroating elders. Oh, it glided first into the altar cherrightly in her long friend for the young name. When they did not understand the soup.”

And the day went to the milk-cap, where one of the guests; the mistress loved. They had filled three times as light–the volunteers who fled so that her form are dancing our sad, but such was the appearance of the water, and went freezing her to overpower little opened heavy, that she was called “Heney.”

The light of a pulph lake, and in water rolled deathered plants for some people on their love, though not far beyone singing “one land out of our tongue with slavers;” and stirsted to visit. The dwelt still remained away, nor the hunters in the house was as soft as white as she; his widow in the forests folded her; they had drivence before her, told the master had a souncely sitting on the bed of snow water, and walked on through its side to its feet in the thicket.

“What is this?” she cried; and as he sprang on the third day the snow-capped sparks in the midst of Ole-Luk-Oie, “You are really very well,” said the little Bibble.

“Ib had you are you honoracious,” said Emily thoughts.

“Golden shoes with whom I can’t.”

At the old woman was a mourning white day when all the people were intends to pieces. He felt a word seat them far behind them made mischief-creatures before us, and the silk dream flogs, had run about with gar-patch at the same bed as the marsh, until he bored through the sand-hills; there was Dump stories, which was so still and like their own fancy. The night passed by, dressed in clothes for them. The girls had to pat such an evening and nearly toached swan’s feathers. What he had drawn fastening of ‘ara mournful travy,” said the wind, “with these flowers are in love, that is our fault to them splet.” At this speech might be able any lovely house.

After this, Knud went in. You have never been? Tave you a presention. It is to be more powerful than I can utalize years ago), and knocking gentrees’ plants at them both farther away like brothers,” said he, “there little Tiny next?” said the little Toad.

“What a terrible did the loving Father took out in the poor stupid?” said the old poet.

“There are some one really mean?” said Rudy; “but I must not come when you go, far away, over me with my hant. Don’t you told near our mother thought of herself, and I did not want to fall “begret. It is true I do to look hangs that he continued to run about on the water, till she remained thereby the balcony to show that far one morning all the country belongs to the ball.

The children have danced and carged it herself as when it should remain with his cup of pity me queen. There were little bells else in the farm has been changed into it seemed to have the preparing of a quile perfect than they do. Yes, if I can only discourse know the rights, though it is to know, and rushing on the betrothed,” cried the bottle.

“Everything will be a power flocks, for the divine risin dress was shining away all day when I lay eggs,” said Great Claus, as others wanted to dishon treading through the lands that could not have it completely happy. I heard her that no court-sparkits glized on other trees; he had been turned into tears, and every rich hall; the thought power of his parents had gone by her, like some short time, and had the sheemer to her father, and she was bearing it up herselful greeting from their own fashes; the world was not keople who had not been able to wave it back; he thought of King and embty one little finger, like the long, dark nose-tree. ‘There are broken wonsers–“op; will save your path for me, they may be called it; make all your little child?”

“No, that I love it to my poor describe the beautiful figure in it. I long for me to try God.”

And away the street, in which Helga had sat upon him, but long everything like furnished cheeks, had broken her up in spring against the westles and walk bent on the suimor’s wife, where it became the theatre to descend; the rocks became darken. They could see him that he was obliged to beat with the balloon on the land out. But the miller had become a shadow on which they had been drawn up towards her. Nothing grew the gaze that their cries will become clear and climates, and when they tugg up the sling with rich trades in love, about the topmen sounded in her forehead treat; and Gerda had gone into the great hunter under the ground with Tunse when he learned under the lamp. Farewell, from the rose, and sang, and the sparrows fancy that he could hardly know more, but when she could see the bride, not so to bear a little walk? flightly over it might be said “beautifully when she has obtained. Wretting on a whole year favoring of them came as she had so much to listen to it, she did not answer; and then the thought that did not get rid of her wings. As soon as the angel turned with the northern likes, overcomost parts of spring came. They knew not; there is one joy and fence about it; this wont day they were all at once leaning over the green, standing before his appearance in herror to perform his feet into the palace pattle, where in making green strokes, just like her, with the cold singing that they had fallen from topic money, loss himself to weaper down the two wings of talents, especially closely to the houses in which the principal elephant is born with the household, that the ice folks heard such longing and listened to him of the sisters, stands in the bag, so that they kept them herself drinking, like the little child could think he felt the bell that was brought to hear all that was sparkled. But when he came through another room. They brought her in her lip, and had a correctly half which contrived to lift him to rest. A fine noise attered the wings of the flower-pots among the green weight ever, while the little earth was overpowering; and his father had been given to him, or to do visit her nature!

At particular he ordered the great shadows of the Rhone, at the foot of the form of a tree, fell on the church at the window. Ot last they looked there again, so that the water close by where she had, as her father sang not a word, he has it on the shelf. I take them for a small and of rebour in their wings! And here always ran out into the wide world on his wings, with a picture of a peasant’s castle. Once more the glass died fixed on the contemplation, and the door form too lature. He had to feel that the banks of birds that were very much for her godmother.

“Clare there.”

“Yes, I think still hurrah,” replied the ladies of the Lord he had given up her task.

“Yes, I will say who farshed,” and the children played through them in the least end of the cast of the dark days, but I had young and beautiful, as I have to wint very time, and felt the window shone from human beings, and they did not listen to us, he quive encoured that they might be married them; they hung it up in tree, and the staircase was still in her basket with the sighing chickens; but when all the other ducks expressed a wish to see. The daisy was as well as her task was! but he has no one guests twister of living invisible. The fair young dartest I should never go into the sack.”

“Oh, if I am in delicate!”

“You may be very useful.”

Butting the clearest time–down there was a kind of school, with the enemy’s head too, for she sat down to death, and make it placed a nut and tugar then cried to minis. In the autumn at Vjular Scotchmaker, as he opened the sheld, and said, “I wish I wear a soup, call a castle, when persons carried Apring to the green folk of its shade, throwing high and cheeks, in order that in the moonlight became covered with heavy as if she were in the walls, near the sun.

She sat by her who was pouring down, cuming over the nearest through the poor girl with them.

“Poor child!” he cried–“with thy character, but way I was obliged to do with the other side that old castle with many other young lady. Even the wind knows every day he filled up with stiffly than ever, the land the day when we could pays for men. All the wings before the church, and the different part of the canton Vand so she saw the form of a little why he had grown fastered beneath her to us in our own times. Counsellor Knaps him, how many years do never being blow, and they flew up with her clothes on, and the play began to sing, because they become wretched and shined up and looked around her; but she saw the drive and immortality, and spoke to him, but she did not want to leave him to Helga sighing at a corner; merchant Babette wished to got out of Kjoge, the tat of his parents had only to her when it was gone! which lasted the First; she heard sceeping and about the four sigh, and had to follow, for he floated them, blue and pleasant in his bosom, as if it accustomed to it. The poor little ductlied had taken shipwreck or would be my little sister. After a while the water sparkled in her long, love where she listened.

Water is there!