محاسبه آمار عبارات یکتا در پایتون — راهنمای کاربردی

1	A critical history of virtual worlds invention
2
3	The job of recounting the specific histories of various virtual 
4worlds developments has already been done, in Japan by Katsura 
5Hattori's _What's Virtual Reality?_ and in the U.S. by Howard 
6Rheingold's _Virtual Reality,_ Myron Krueger's _Artificial Reality 
7II,_ and other books to be published this year.  I want to emphasize 
8the individualistic character of the invention that has taken place so 
9far, which may explain why our technology is less than we would 
10have it be.
11
12	Clearly, many inventors were inspired by science fiction 
13stories, in which people traveled through space and time, either 
14physically "teleporting" their bodies or sending their thoughts 
15around via telepathy.  Vannevar Bush was perhaps the first modern 
16computer scientist to conceive of knowledge as a medium through 
17which one might travel by machine.  His "Memex" was a fantasy 
18computer that would put all knowledge at the disposal of its user in 
19multimedia form,   Now Vannevar Bush's dream of universal access 
20to knowledge has become international.
21
22	In the U.S., in the 1960s, Ivan Sutherland started experiment-
23ing with stereoscopic images created by a computer, to build a "data 
24field."  Around the same time, Myron Krueger, another American, was 
25using video cameras and other techniques to reverse the flow, put-
26ting the user "into" the computer terminal and merging him or her 
27with the images on the screen.  And, in the next decade, Thomas 
28Furness began directing the "Super Cockpit" project for the U.S. Air 
29Force, a completely pilot-attuned ensemble (complete with virtual 
30world projectors) that the pilot would wear.  While these eminences 
31knew of each other, their work did not coalesce but continued in dis-
32tinctly different directions:  Sutherland, toward flat-screen simula-
33tors; Krueger, deeper into art and media environments; and Furness, 
34toward continuing to refine data-presentation and manipulation.
35	
36	The popularization of virtual worlds occurred with the simpli-
37fication of a stereoscopic, head-mounted data display with position-
38sensors by NASA; and the subsequent commercialization of a similar 
39display complete with world-design software, the "Eyephones" and 
40"Body Electric/Swivel 3D" by VPL Research, in Redwood City, Cali-
41fornia.  With the appearance of these systems, and later the 3D 
42sound unit, the Convolvatron, marketed by Crystal River Engineering, 
43virtual worlds took off in the press and popular imagination.  Auto-
44desk, of Sausalito, California, announced its work on Cyberspace, a 
453D CAD program.  Cyberspace begat Sense8, also of Sausalito, a 
46small firm working on a Sun workstation platform rather than the 
47traditional, more powerful Silicon Graphics computers that had been 
48the tradition until then.  For awhile it looked as if the Mattel Power-
49glove, a derivative of VPL's Dataglove developed by A.G.E., in New 
50York, might take virtual worlds into the exploding Nintendo game 
51market, but this did not occur.
52
53	However, although work in virtual worlds was going on in many 
54places by the 1990s, almost all of this activity was completely 
55independent and uncoordinated.  Developments and inventions would 
56usually become known within the virtual worlds community only 
57after their introduction at one of the computer professional conven-
58tions or trade shows, and even so inventors liked to hold onto secret 
59code or hardware tricks to keep their pint-sized corporate empires 
60intact.  This tradition of individuality and secrecy is only slowly 
61being eroded by professional and social communications.  Unfortu-
62nately, it is no longer the legacy of small firms; today, many larger 
63firms practice the same self-serving tactics, to the disadvantage of 
64our field.

1"""Print most frequent N-grams in given file.
2Usage: python ngrams.py filename
3Problem description: Build a tool which receives a corpus of text,
4analyses it and reports the top 10 most frequent bigrams, trigrams,
5four-grams (i.e. most frequently occurring two, three and four word
6consecutive combinations).
7NOTES
8=====
9I'm using collections.Counter indexed by n-gram tuple to count the
10frequencies of n-grams, but I could almost as easily have used a
11plain old dict (hash table). In that case I'd use the idiom
12"dct.get(key, 0) + 1" to increment the count, and heapq.nlargest(10)
13or sorted() on the frequency descending instead of the
14counter.most_common() call.
15In terms of performance, it's O(N * M) where N is the number of words
16in the text, and M is the number of lengths of n-grams you're
17counting. In this case we're counting digrams, trigrams, and
18four-grams, so M is 3 and the running time is O(N * 3) = O(N), in
19other words, linear time. There are various micro-optimizations to be
20had, but as you have to read all the words in the text, you can't
21get much better than O(N) for this problem.
22On my laptop, it runs on the text of the King James Bible (4.5MB,
23824k words) in about 3.9 seconds. Full text here:
24https://www.gutenberg.org/ebooks/10.txt.utf-8
25I haven't done the "extra" challenge to aggregate similar bigrams.
26However, what I would do to start with is, after calling
27count_ngrams(), use difflib.SequenceMatcher to determine the
28similarity ratio between the various n-grams in an N^2 fashion. This
29would be quite slow, but a reasonable start for smaller texts.
30This code took me about an hour to write and test. It works on Python
312.7 as well as Python 3.x.
32"""
33
34import collections
35import re
36import sys
37import time
38
39
40def tokenize(string):
41    """Convert string to lowercase and split into words (ignoring
42    punctuation), returning list of words.
43    """
44    return re.findall(r'\w+', string.lower())
45
46
47def count_ngrams(lines, min_length=2, max_length=4):
48    """Iterate through given lines iterator (file object or list of
49    lines) and return n-gram frequencies. The return value is a dict
50    mapping the length of the n-gram to a collections.Counter
51    object of n-gram tuple and number of times that n-gram occurred.
52    Returned dict includes n-grams of length min_length to max_length.
53    """
54    lengths = range(min_length, max_length + 1)
55    ngrams = {length: collections.Counter() for length in lengths}
56    queue = collections.deque(maxlen=max_length)
57
58    # Helper function to add n-grams at start of current queue to dict
59    def add_queue():
60        current = tuple(queue)
61        for length in lengths:
62            if len(current) >= length:
63                ngrams[length][current[:length]] += 1
64
65    # Loop through all lines and words and add n-grams to dict
66    for line in lines:
67        for word in tokenize(line):
68            queue.append(word)
69            if len(queue) >= max_length:
70                add_queue()
71
72    # Make sure we get the n-grams at the tail end of the queue
73    while len(queue) > min_length:
74        queue.popleft()
75        add_queue()
76
77    return ngrams
78
79
80def print_most_frequent(ngrams, num=50):
81    """Print num most common n-grams of each length in n-grams dict."""
82    for n in sorted(ngrams):
83        print('----- {} most common {}-grams -----'.format(num, n))
84        for gram, count in ngrams[n].most_common(num):
85            print('{0}: {1}'.format(' '.join(gram), count))
86        print('')
87
88
89if __name__ == '__main__':
90
91    start_time = time.time()
92    with open('C:/Users/m.jaderyan/Desktop/Text.txt') as f:
93        ngrams = count_ngrams(f)
94    print_most_frequent(ngrams)
95    elapsed_time = time.time() - start_time
96    print('Took {:.03f} seconds'.format(elapsed_time))

1----- 15 most common 2-grams -----
2virtual worlds: 21
3in the: 17
4at the: 9
5university of: 9
6of the: 8
7the virtual: 8
8the university: 8
9the world: 7
10the computer: 7
11virtual world: 7
12and the: 6
13on the: 5
14of virtual: 4
15it is: 4
16in three: 4
17
18----- 15 most common 3-grams -----
19the university of: 8
20the virtual worlds: 4
21of virtual worlds: 3
22the virtuality paradigm: 3
23the virtual world: 3
24the physical world: 3
25at the university: 3
26the u s: 3
27research is taking: 3
28is taking place: 3
29virtual worlds technology: 2
30of this technology: 2
31around the world: 2
32the world and: 2
33within the computer: 2
34
35----- 15 most common 4-grams -----
36at the university of: 3
37research is taking place: 3
38in the fields of: 2
39the virtual world can: 2
40the university of north: 2
41university of north carolina: 2
42in three or more: 2
43three or more dimensions: 2
44in the u s: 2
45is taking place in: 2
46the virtual worlds consortium: 2
47in japan and north: 2
48japan and north america: 2
49virtual worlds technology promises: 1
50worlds technology promises to: 1
51
52----- 15 most common 5-grams -----
53the university of north carolina: 2
54in three or more dimensions: 2
55research is taking place in: 2
56in japan and north america: 2
57virtual worlds technology promises to: 1
58worlds technology promises to greatly: 1
59technology promises to greatly expand: 1
60promises to greatly expand both: 1
61to greatly expand both the: 1
62greatly expand both the numbers: 1
63expand both the numbers of: 1
64both the numbers of persons: 1
65the numbers of persons who: 1
66numbers of persons who use: 1
67of persons who use computers: 1