creating custom dataset from huggingface dataset

Sample code can be found here.  

You can create dataset by using the the python module: dataset. If you don't have it, please run 

pip install dataset

Next, ensure you have a file with content called my_train.txt and my_test.txt. Both file needs content otherwise it won't work

from datasets import load_dataset

dataset = load_dataset('text', data_files={'train': ['my_train.txt'],

'test': 'my_test.txt'})

To access these dataset, you can either loop or directly reference: As you can see here, we have train and test

> print(dataset['train'][0])

{'text': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'}

> print(dataset['test'][0])

{'text': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'}

Other supported format are cvs, json     

## loading json dataset

dataset = load_dataset('json', data_files={'train': ['my_train_json.json'], 'test': 'my_train_json.json'})

print(dataset)

print(dataset['test'][0])

To show the output of your json file 

> dataset['test'][0]

{'label': 0, 'text': 'bbbbbbbbbbbbbbbbbbbb'}

As you can see from here, we are using the same technique in one of huggingface colab

from datasets import load_dataset

dataset = load_dataset("yelp_review_full")

dataset["train"][100]

Sample output:

----------------------------------------------------------------------

{'label': 0, 'text': 'My expectations for McDonalds are t rarely high. But for one to still fail so spectacularly...that takes something special!\\nThe cashier took my friends\'s order, then promptly ignored me. I had to force myself in front of a cashier who opened his register to wait on the person BEHIND me. I waited over five minutes for a gigantic order that included precisely one kid\'s meal. After watching two people who ordered after me be handed their food, I asked where mine was. The manager started yelling at the cashiers for \\"serving off their orders\\" when they didn\'t have their food. But neither cashier was anywhere near those controls, and the manager was the one serving food to customers and clearing the boards.\\nThe manager was rude when giving me my order. She didn\'t make sure that I had everything ON MY RECEIPT, and never even had the decency to apologize that I felt I was getting poor service.\\nI\'ve eaten at various McDonalds restaurants for over 30 years. I\'ve worked at more than one location. I expect bad days, bad moods, and the occasional mistake. But I have yet to have a decent experience at this store. It will remain a place I avoid unless someone in my party needs to avoid illness from low blood sugar. Perhaps I should go back to the racially biased service of Steak n Shake instead!'}