Corpus zilankhulo ndi chiyani?

Zaka zingapo zapitazo kuti automate kufufuza zinenero, asayansi ndinkaona ngati sindingazikwanitse. ntchito chidachitika ndi dzanja, amaitana ambiri ophunzira, pali yokulirapo imabwera "osasamalira" kulakwitsa, ndipo ambiri Chofunika - zonse izi anatenga nthawi yaitali.

Ndi chitukuko cha makompyuta wakhala zotheka kuti kafukufuku pa dongosolo la udzakwaniritsidwa mofulumira, ndipo lero mmodzi wa mayendedwe kwambiri zingamuthandize mu kuphunzira chinenero ndi corpus zilankhulo. Mbali yake yaikulu ndi ntchito yambiri mudziwe malemba, mfundo ku Nawonso achichepere limodzi, m'njira yapadera ndipo akutchedwa thupi, n'kuchiumba.

Pakali pano, pali nyumba zambiri analengedwa ndi zolinga zosiyana pamaziko a zinenero zinthu zosiyanasiyana zinatenga ku miyandamiyanda makumi mabiliyoni mayunitsi lexical. malangizowo chimatengedwa ngati adzapeza ndipo imasonyeza patsogolo kwambiri kwa zolinga ntchito ndi kafukufuku. Akatswiri, njira imodzi kapena zimabwera wina ndi chinenero zachilengedwe, izo tikulimbikitsidwa kuti timudziwe bwino thupi la malemba osachepera mlingo zofunika.

History of corpus zilankhulo

Mapangidwe kukuchititsa zimenezi chifukwa kulengedwa kwa United States pa Brown thupi kumayambiriro 60 ntchentche Mu zaka zapitazo. Zosonkhanitsira zikuphatikizapo malemba onse miliyoni 1 mitundu mawu, ndipo lero thupi la kukula uyu adzakhala kwathunthu uncompetitive. Ichi ndi chifukwa cha mlili wa mayendedwe a chitukuko cha makompyuta, komanso zofuna kukula kwa zinthu zatsopano kafukufuku.

Mu 90s ndi zilankhulo corpus anatuluka mu mwambo wonse ndi ufulu gulu la malemba akokedwa ndi ankatha m'zinenero zambirimbiri. Mu nyengo iyi analengedwa Mwachitsanzo, boma la Britain National Corpus 100 miliyoni mphatso.

Ndi chitukuko cha m'dera limeneli la zilankhulo, mabuku lemba akukhala zambiri (ndi kukwaniritsa mabiliyoni dictionary mayunitsi), ndi kamangidwe likupitirirabe osiyanasiyana. Pakali pano, danga Internet angapezeke mitembo olembedwa ndi chinenero, zinenero, ndi kuphunzira wokonda mabuku luso kapena maphunziro, komanso mitundu ina yambiri.

Kodi nyumba

Mitundu thupi zilankhulo thupi zingakhalepo pa zifukwa zingapo. Intuitively, maziko gulu kungakhale chinenero malemba (Russian, German), mwayi mode (lotseguka gwero, anatseka, malonda), ndi mtundu wanyimbo wa gwero (zopeka, zopelekedwa, maphunziro, utolankhani).

Chidwi njira amapanga zipangizo za amalankhula. Popeza kujambula dala mawu amenewa kuti amalola yokumba kwa anayankha, ndi chuma chifukwa sichikanakhoza kutchedwa "mowiriza" masiku ano zilankhulo corpus wapita njira zina. A Wodzipereka okonzeka ndi maikolofoni, ndipo masana azimva mbiri ya zokambirana zonse, imene nawo. Anthu padziko, ndithudi, asadziwe kuti m'kupita kwa kucheza tsiku lililonse kumathandiza kuti chitukuko cha sayansi.

Analandiranso mbiri kusungidwa mu Nawonso achichepere ndipo ali limodzi ndi kusindikizidwa mtundu lemba mawu olembedwa. Choncho, imatheka chizindikiro patsambali anafunika kulenga m'kamwa tsiku kulankhula nyumba.

ntchito

Pakufunika kutero ntchito chinenero, ndipo mwina ntchito nyumba malemba. Njira ntchito alumikizana mu zilankhulo zingakhale:

• Kukonza ndondomeko kudziwa chinsinsi, chimagwiritsidwa ntchito ndale ndi malonda kukhala ndi njira yolongosoka mayankho zabwino ndi zoipa adzaponye chisanko ndi makasitomala, motero.
• Kulumikiza dongosolo mudziwe kuti madikishonale ndi omasulira bwino ntchito yawo.
• A zosiyanasiyana kufufuza zimene zimachititsa kuti timvetse bwino unit chinenero, mbiri ya chitukuko ndi ulosi wa kusintha posachedwapa.
• Development wa mudziwe machitidwe katengedwe zochokera morphological, syntactic, zamalankhulidwe ndi zina mbali.
• Kukhathamiritsa wa machitidwe osiyana zinenero ena.

Ntchito ya nyumba

zofanana gwero mawonekedwe ndi mmene injini zofufuza zanu, ndi kukuchititsa wosuta kulowa mawu kapena kuphatikiza mawu kufufuza m'munsi zambiri. Kuwonjezera kupanga kafufuzidwe enieni kugwiritsa ntchito Baibulo patsogolo, amene amalola kupeza mfundo kalembedwe pa muyezo chilichonse chinenero.

Kufufuza m'munsi zingakhale:

• umembala wa gulu linalake mbali ya kulankhula;
• mbali kalembedwe;
• manenedwe;
• stylistic ndi maganizo ochekenera.

Mukhozanso kuphatikiza i apeze ndondomeko ya mawu, mwachitsanzo, kupeza nthawi zonse wa mneni mu pakali pano, munthu woyamba mmodzi, amene akubwera pambuyo pa anawagwiritsira "mu" ndi liwu lofotokozera kapena lokamba zinachitikira accusative. Yothetsera zimenezi ntchito yosavuta amatenga wosuta masekondi angapo ndipo amafuna chabe kudina mbewa angapo m'minda mwachindunji.

The ndondomeko ya kulenga

Kufunafuna yokha angathe kuchitidwa pa subcorpus onse ndi chimodzi mwachindunji osankhidwa, malingana ndi zosowa kukwaniritsa cholinga chinachake:

1. Sitepe yoyamba ndi kufotokoza chimene malembawa maziko choncho. Zolinga zothandiza kuti nthawi zambiri ntchito journalistic, Nkhani, Intaneti ndemanga. The ntchito kafukufuku ndi ntchito zosiyanasiyana mitundu phukusi, koma lemba ayenera anasankha monga ena mumafanana.
2. Zosonkhanitsira chifukwa malemba pansi pretreatment, pali kuwukonza zolakwika, ngati aliyense, lokonzedwa ndi malongosoledwe bibliographic ndi owonjezera zinenero lembalo.
3. Likuchotsedwa onse mudziwe sanali kalembedwe: kukonza zithunzi, zithunzi, matebulo.
4. Ndi Kugawilidwa kwanthaka wa mphatso, amene amangoona kulankhula, chifukwa processing zina.
5. Pomaliza, inatenga morphological, syntactical ndi kulemba ena analandira kuchuluka kwa zinthu.

The chifukwa cha wotuluka zonse zotentha ndi dongosolo syntactic ndi anagawira m'menemo ndi kuchuluka kwa zinthu, omwe imafotokozedwa mbali ya mawu, kalembedwe ndi nthawi zina, zotsatira zamalankhulidwe.

Mavuto polenga nyumba

Nkofunika kuzindikira kuti sikokwanira kuika pamodzi ndi mawu amenewa kapena ziganizo thupi. Pa dzanja limodzi, gulu la malemba ayenera kukhala anzeru, ndiye akuimira mitundu yosiyanasiyana ya malemba mu kufanana ena. Pa zina - nkhani za mpanda ayenera spaced m'njira yapadera.

Vuto loyamba ndi anathana ndi mgwirizano Mwachitsanzo, mu deta kumaphatikizapo 60% ya m'mabuku, 20% ya mabuku, kuchuluka zina amaperekedwa chifaniziro olembedwa a chinenero chimene chinkalankhulidwa, malamulo, ntchito ya sayansi, etc. wangwiro Chinsinsi moyenera thupi lero kulibe ...

Funso lachiwiri, za kamangidwe okhutira, kuthetsa zovuta. Pali zochitika zapadera ndi ma aligorivimu ntchito basi mukulemba malemba, koma iwo musataye chifukwa wangwiro, angachititse disruptions ndipo amafuna Buku rework. Mwayi ndi mavuto polimbana ndi vutoli zafotokozedwa mwatsatanetsatane mu pepala V. P. Zaharova a zilankhulo corpus.

Text chizindikiro patsambali akuyendera misinkhu zingapo, zimene ife Lembani m'munsimu.

morphological tagging

Kusukulu, tiyenera kukumbukira kuti chinenero cha Chirasha, pali madera osiyana a mawu, ndipo aliyense wa iwo ali ndi makhalidwe yake. Mwachitsanzo, mneni ali m'magulu a ndingaliro ndiponso nthawi imene palibe chachikazi. wokamba mbadwa ndilipo m'tali manauni ndi kasewenzetsedwe ka mau conjugate, koma awonetse thupi la 100 miliyoni. mphatso ntchito yamanja sizigwira ntchito. ntchito zonse zofunikira angathe kudzapereka kompyuta Komabe, izi ziyenera kuphunzitsidwa.

Morphological tagging, kompyuta ayenera "kumvetsetsa" liwu lililonse ngati mbali zina za kulankhula ndi ena mbali kalembedwe. Popeza Russian (ndi zilankhulo zina) ukugwira angapo malamulo okhazikika, n'zotheka kuti amange ndondomeko zodziwikiratu kwa kusanthula morphological, pakuika galimoto kwa chiwerengero cha ma aligorivimu. Komabe, pali mwaiwala zimene malamulo, komanso zinthu zosiyanasiyana complicating. Chifukwa, ukonde kompyuta kusanthula lero ali kutali kwambiri, ndi ngakhale 4% zolakwa chipereka mtengo wa mln 4. Words pa thupi la 100 miliyoni. Mayunitsi, napempha Buku rework.

Mwatsatanetsatane m'buku akufotokoza vuto Zaharova V. P. "Corpus zilankhulo".

syntactic Zamkati

Parsing kapena parsing - ndondomeko kumene kumatsimikizira ubale wa mawu m'sentensi. Ntchito ya ma aligorivimu ndi n'zotheka kudziwa lemba la mutu, mkhalidwe, zowonjezera, ankasinthana angapo wa kulankhula. Tikupeza amene mawu ndi ndondomeko zazikulu, ndi omwe - yozungulira, ife angathandize kuchotsa mfundo malemba ndi kuphunzitsa makina apereke atamupempha kufufuza okha mfundo chidwi ife.

Mwa njira, injini ano kufufuza ntchito kupereka manambala enieni m'malo malemba yaitali poyankha mafunso zogwirizana monga "angati zopatsa mphamvu mu apulo" kapena "mtunda kuchokera Moscow ku St. Petersburg." Komabe, kumvetsa ngakhale ndizosowa ndondomeko anafotokoza bwino kufunsa "Mawu Oyamba kwa Corpus zilankhulo" kapena zina zofunika phunziro.

zamalankhulidwe chizindikiro patsambali

The manenedwe a mawu - ndi, mawu osavuta, tanthauzo. njira chimagwiritsidwa ntchito pa kusanthula zamalankhulidwe a Tags mawu Attribution kuwalitsa wake ndiwe wa ya siyana zamalankhulidwe ndi subcategories. Mfundo zimenezi ndi zamtengo wapatali kwa optimizing ma aligorivimu tisanthule malemba kamvekedwe, basi summarization ndi njira zina ntchito za zilankhulo corpus.

Pali zingapo "muzu" wa mtengo, woimira mawu umboni ndi manenedwe lonse. Monga nthambi ya mfundo mtengo anapanga, munali ndi kutchula zinthu lexical. Mwachitsanzo, mawu akuti "cholengedwa" akhoza kugwirizana ndi mfundo monga "anthu" ndi "ziweto". Mawu oyamba adzapitiriza nthambi mu kudzinenera osiyana, mawu zimakhalira pachibale, fuko, ndipo chachiwiri - pa makalasi ndi mitundu ya nyama.

Ntchito zambiri kachitidwe katengedwe

Madera a ntchito zilankhulo corpus kuphimba minda osiyanasiyana ntchito. Housings ntchito yokonza ndi kuwukonza otanthauzira, kulenga machitidwe yodzichitira translation, annotating, retrieving mfundo kuzindikira kamvekedwe ndi zina processing malemba.

Komanso, chuma amenewa ntchito mwakhama kuphunzira zilankhulo dziko ndi njira ya ukugwirira ntchito ya chinenero ambiri. Kupeza mabuku lalikulu mudziwe chisanadze okonzeka facilitates kafukufuku mofulumira ndi mwatsatanetsatane za zochita za zilankhulo za chitukuko komanso kusintha khola mapangidwe neologisms kulankhula liwiro amayamikira mayunitsi lexical ndi ena.

Popeza ntchito ndi ndalama lalikulu deta amafuna zokha, lero pali zimachitika pafupi pakati pa kompyuta ndi corpus zilankhulo.

Russian National Corpus

Pankhaniyi (chidule NKRYA) zikuphatikizapo angapo subcorpus, kulola kugwiritsa ntchito ndalama chifukwa zosiyanasiyana ntchito.

zipangizo mu Nawonso achichepere ndi zidzasankhidwa NKRYA:

• mabuku mu 90s TV 'ndi 2000s, zoweta ndi achilendo;
• Polemba mawu;
• aktsentologicheski amalembedwa malemba (i.e., zipsera za maganizo);
• kulankhula chilankhulo;
• ndakatulo;
• Zida ndi kulemba syntactic ndi zina.

Kachitidwe mudziwe mulinso Subcorpus matanthauzidwe yomweyi ntchito ku Russian mu English, German, French ndi ambiri m'zinenero zina (ndi mosemphanitsa).

Komanso Nawonso achichepere pali gawo la malemba mbiri, kuimira mawu olembedwa mu Russian mu nthawi zosiyana za chitukuko. Palinso maphunziro thupi, zomwe zingakhale zothandiza kwa anthu achilendo lophunzira Chirasha.

Russian National Corpus limapangidwa 400 miliyoni mayunitsi lexical, ndiponso m'njira zambiri patsogolo pa mbali yaikulu ya zinenero za ku Ulaya matupi.

ziyembekezo

Ndipotu mokomera kuzindikira kukuchititsa zimenezi kupezeka kwa kulonjezedwa zasayansi corpus zilankhulo ku mayunivesite Russian, komanso akunja. Ntchito ndi kafukufuku mu chimango cha mfundo ndi kufufuza izi chuma kumatanthauza chitukuko cha zinthu zina m'munda wa umisiri mkulu ka funso kuyankha, koma lafotokozedwa pamwamba.

pofuna kukwaniritsa corpus zilankhulo unaloseredwa paliponse, kuyambira luso mawu a kukhazikitsa ma aligorivimu latsopano kuti konza njira kufunafuna ndi processing mudziwe mphamvu makompyuta, RAM kwambiri, ndipo kwa ogula, chifukwa owerenga ali ndi njira kugwiritsa ntchito mtundu wa gwero mu tsiku lawo moyo ndi ntchito.

pomaliza

Mu zaka za m'ma lomaliza 2017 ankaoneka kutali mtsogolo, pamene ndege Pakuyenda chilengedwe ndi makina kuchita ntchito zonse kwa anthu. Ndipotu, sayansi lili ndi "mawanga woyera" ndipo anayesa ofunitsitsa kuyankha mafunso a anthu kwa zaka akusokoneza. Mafunso ukugwirira ntchito ya chinenero pano zatenga malo a ulemu, ndi nduna ndi zowerengera zilankhulo chingatithandize kuwayankha.

Processing kukukhala lalikulu deta angazindikire dongosolo, poyamba kufikako, kuneneratu chitukuko mbali chinenero yeniyeni younikira mapangidwe mawu mu nthawi pafupifupi chenicheni.

Pa mlingo machitidwe, ndi m'makola mwawo lonse Tingaone Mwachitsanzo, ngati chida kuthekera kuti aone maganizo pagulu - pa Internet ndi zonse kusinthidwa tsiku maziko malemba ena analengedwa ndi ogwiritsa chenicheni ichi ndemanga ndi ndemanga, ndi nkhani komanso zinthu zina zambiri kulankhula.

Komanso, ntchito ndi matupi limathandiza kuti chitukuko cha hardware chomwecho, kuti zimakhudzidwa mudziwe katengedwe, ndife bwino utumiki "Google" kapena "Yandex", makina yomasulira, otanthauzira amagetsi.

Timadziwa kuti amanena kuti corpus zilankhulo amapanga okha njira woyamba, ndipo posachedwapa adzasangalala.