U Aez@sddlZddlZddlZddlZddlZddlZddl Z ddl m Z m Z ddl Z de jd<ddlmZddlmZddlmZddlmZdd lmZmZdd lmZdd lmZd d ZddZddZ ddZ!ddZ"ddZ#ddZ$ddZ%ddZ&ddZ'd d!Z(d"d#Z)d$d%Z*d&d'Z+dWd(d)Z,d*d+Z-dXd,d-Z.d.d/Z/d0d1Z0dYd2d3Z1dZd4d5Z2d6d7Z3d8d9Z4d[d;d<Z5d=d>Z6d?d@Z7d\dBdCZ8dDdEZ9dFdGZ:d]dIdJZ;dKdLZdQdRZ?d^dSdTZ@dUdVZAdS)_N)Markdowndisplayz$/home/nano/.pybliometrics/config.iniZPYB_CONFIG_FILE)AbstractRetrieval) FreeProxy)Works)SequenceMatcher) get_templaterender_to_string)Context) HttpResponsec Csddlm}m}ddl}ddlm}|D]}|jj|jd}|j dddgdd }| d |_ d |_ d |_ d |_d |_d|_d|_d|_d|_d|_d|_|d}ddg} |} | D]} | || qtdd} | | | W5QRXd|_|tddq(dS)Nr) PdfFileReader PdfFileWriterCVidZpdflatexzcv.texz-interaction=nonstopmodez"/var/www/nanotud/uploads/cv_files/)cwd4uploads/cv_files/auto_include/papers_auto_django.texzuploads/cv_files/cv.logzuploads/cv_files/cv.pdfzuploads/cv_files/cv.texz.uploads/cv_files/auto_include/conferences_autoz*uploads/cv_files/auto_include/patents_auto1uploads/cv_files/auto_include/papers_auto_sel.texz'uploads/cv_files/auto_include/phds_autoz(uploads/cv_files/auto_include/talks_autoz'uploads/cv_files/auto_include/rest_autoz(/var/www/nanotud/uploads/cv_files/cv.pdfz./var/www/nanotud/uploads/cv_files/cv_short.pdfwbzuploads/cv_files/cv_short.pdfz/data/uploads/cv_files/cv.pdfz$/data/uploads/cv_files/cv_backup.pdf)ZPyPDF2r r subprocessinternal.modelsrobjectsgetrPopenwaitZ pub_includelog_filepdf_filetex_filepublication_includeconferences_includepatents_includepapers_selected_include phds_include talks_include rest_includeZaddPageZgetPageopenwriteclose pdf_shortsaveshutilcopy2) querysetr r rrqsentryppdfpagesZ pdfWriterZpage_numfr5#/var/www/nanotud/modules_webpage.py pdflatex_cvs:   r7c Csddlm}ddlm}d|jjddi}td|dd }td d }||W5QRXd|jjd d i}td|dd }tdd }||W5QRX|D](}|jj |j d}d|_ d|_ | qdS)Nr PublicationrZpublication_list CunibertiZauthor_list__icontainszpublication_export.textex)usingzE/var/www/nanotud/uploads/cv_files/auto_include/papers_auto_django.texwT) cvselectedzpublication_export_selected.texzB/var/www/nanotud/uploads/cv_files/auto_include/papers_auto_sel.texrrr)research.modelsr9rrrfilterr r'r(rrr r#r+)r.r9rcontextcontentZ static_filer/r0r5r5r6create_publication_includeAs    rDcCs,ddlm}|j}|D] }|qdS)NrMember)rrFrrAr+)r.rFmembersmemberr5r5r6 set_slug_2Ys    rIcCs,ddlm}|j}|D] }|qdSNrr8r@r9rrAr+)r.r9pubspubr5r5r6 set_slug_3`s    rNcCs,ddlm}|j}|D] }|qdS)Nr)Event)external.modelsrOrrAr+)r.rOZevsnr5r5r6 set_slug_4gs    rRcCs,ddlm}|j}|D] }|qdSrJrK)r.r9rLZsemr5r5r6set_slug~s    rSc Cst|dkrv|D]^}td|jd|jdt|j}tt|}t }|dD]}|j |dddd }q\|d|d <||d  }|d  d |d <|jdkr|}n||d t |jk}tt|dt|dkr|jj|jd}d|_|dSddlm}t|j|jD]F\} } t| rDd} nt| } || | |jddd} | q*qntddS)NrzFind publications for z from z in Google Scholar publicationsbibTF ignore_indexsort google_pubtitlestrpub_yearz publications foundrrz&No publication found in Google Scholar)Add_Publication,)rZyearauthorzPlease select only one entry)lenprintr`r_ scholarly search_authorfillnextpd DataFrameappendisnullastyper[rrrcommentr+r@r]zipr\rZisnaintsplit) r.r/ search_queryr` pub_googlerMr\r]r0r_rZZnew_pubr5r5r6add_dois_scholar_infos8       rscCs(|dkr dSd}|ddks\d|ddks\d|ddks\d|ddks\d |dd kr|d d kr|||d d dd}nD|dd kr|||ddd}n|||ddd}n0d|ddkr||d|d}n||}|dd|dd|dd|S)Nr© Copyrightc○ COPYRIGHTThis journal isWeinheim Switzerland .rz z z!Abstract)findreplace)txtstringr5r5r6clean_abstracts L    rcCs0d}|D]"}t|dkr"|d7}q||7}q|S)Nrti -)ord)Zinp_strnewc0r5r5r6check_unicode_chars    rcCs$ddlm}m}ddlm}|D]}d}|jj|jd}t|j  d}|dksb|dksb|dkr|j dkrt dt |j |j}|dkst|rd |_|d }n||_ ||jj|d }d \ } } } } } }}}}}}|dkr|dkrzt}| |}Wnt |d d}YnXzt|d d}Wnt |dd}YnX|dkrx|dkrxd|_|qtt||}|dkrxt|j } t| } tt||} |j} |jdkr|j}t|}|jdkr|j}t|}z|j !dd }Wn(z |j"}Wnd}YnXYnXz|j#} | dd} Wnd} d} YnXz |j$}Wnd}YnX|dkr| dkrt%||d} | dkr|dd} | dkrzt|d} Wn YnX|dkr|d}| dkrBz>t&|} | !ddd| !ddd| !dd} Wnd} YnX|dkrfz |d}Wn YnX|dkrz |d}Wn YnX|dkrz|dd}Wn YnX|dkrz|dd}Wn YnX| dkrzt'|dddd} Wn YnX|dkr*|}|dkr*d }||d| | | | | ||||||d d!}||dkrhd"|_n|dkrzd#|_nd$|_d |_(|n:|d krd%|_d|_(n"t d&d'|_d |_)d|_(||j*}|D]D}|jjdd(j|j+dd)|j,d*}|D]}|j-.|qqq dS)+Nr)r9r]rEFrrtNoneZherezerozNo DOI found for titleT)Z doi__iexact) NNNNNNNNNNNz not found in Crossref.)Zrefreshz8 not found in Scopus. API key and permission correct?(2)zODOI neither found in Scopus nor in Crossref. Please do check if DOI is correct.ritemrZvolumetyperr~rpageissueshort-container-titlezcontainer-titlecreated date-partsZXXXX)doiapprovedr_rZ author_listpub_dater open_accessabstractr journal_shortjournalpub_type chair_pubzInformation taken from Scopus.z Information taken from Crossref.z+Information taken from Scopus and Crossref.z No DOI found.zpublication already therezpDOI already in publication list. If you want to add it again, please delete the corresponding publication first.)r. r;)/r@r9r]rrFrrrr[rstriprZrbfind_doi_for_titler`rgrnrlr+rAcountrrr get_abstracthtmlunescaperformat_authors_scopusrsourcetitle_abbreviationZpublicationNameZ pageRangerp startingPage coverDateZopenaccessFlagformat_authors_crossrefpub_date_crossroadded doi_in_pubsall given_name last_name publicationadd)r.r9r]rFr/Zkakar0rnumrZauthorsrrr_rrZtypeenumberrrworksitem_CR item_scopusrrMZiddirLpir5r5r6add_publication_infos                        6                &rc Cst|d}|}W5QRX|ddd}|D]t}|d}i}|D]"}|dd}z|dd kr`|d kr`d |d d kr`d |d dkr|d d|d <nd|kr|jdddd|d<nd|kr|jdddd|d<nd|jddd}t|dkr&|jddd}t|dkrHtj||d <n|d||d <WqLYqLXqL|ikr4z| j|dd} Wq4t j |d gd} Yq4Xq4| tj} | S)u* Reads publication from database given by the format of the webpage. Parameters ---------- file_name : name of the database file Structure: Two columns (1 = property, 2 = value ) Entries are separated by 'enditem ##################################' Example: '########################################################################## AG GC cvselected no firstlast yes #filename cover ../pubs/reprints/covers/2020_Krok.jpg year 2020 title Modification of titanium implants using biofunctional nanodiamonds for enhanced antimicrobial properties authors Emilia Krok, Sascha Balakin, Jonas Jung, Frank Gross, Jörg Opitz, G. Cuniberti pubdate 2020.03.02 type regular reference Nanotechnology 31, 205603 (2020) DOI 10.1088/1361-6528/ab6d9b/ abstract The present study describes a novel antimicrobial surface using anodic oxidation of titanium and biofunctional detonation nanodiamonds (ND). ND have been loaded with antibiotics (amoxicillin or ampicillin) using poly (diallyldimethylammonium chloride)(PDDA). Successful conjugation with PDDA was determined by dynamic light scattering, … enditem ########################################################################### ...' Returns ------- DataFrame Containing all publication entries from 'file_name'. Each row contains one publication and columns its values. If one property is not set in file_name, its value is set to 'np.NaN'. renditemrr  r~#rtrrr journalfullname maxsplit arxivpasswdTrWindex) r'readrprrranpNaNrirgrhfillna) file_namefilerCZall_publicationsarticle attributesdictioattrpdfr5r5r6read_publicationsts>$     * rcCs|d}i}|D]}|dd}z|ddkr|dkrd|ddkr|jdd d }t|d kr|jd d d }t|d krtj||d<q|d||d<n|d||d<WqYqXq|S) Nrrrr~rrtrrrr)rprrarrr)r0rrrrpr5r5r6 entry_to_dicts   $  rc Cst|d}|}W5QRX|d}|D]D}t|}|ikr,z|j|dd}Wq,tj|dgd}Yq,Xq,|tj }||d |d @j j }|j |dd |S) NrrTrrrZvornameZnachname)inplace)r'rrprrirgrhrrrrnrvaluesdrop)rrrCZ all_entriesr0rrindr5r5r6read_addressess    rc Csd}t|d}||jdddd|D]t\}}||}|D]J\}}t|dkr~||dt|dqN||dt|dqN||q0W5QRXdS) N:enditem ################################################# r>rrrr )r'r(rpiterrowsrnitemsrar[) rr seperatorrrrowZto_writeZindex_wvaluer5r5r6write_addressess  rcCsd||}tt|dS)Nz{})formatrr)rcolorZcolorstrr5r5r6printmds rc Csd}d}d}g}|jD]}|j|j}t|s@t|d}t|sR|dkrt|j|j}||j|df<t|st dt|d|j|j|d7}t|r| t j |j|j dkr|j|j d krt d |j|jd |j|j d qt|}|dkr$| ||d7}qt|j|j}t|sJ||krX| t j qt d |d||d7}||j|df<t|}|dkr| ||d7}q| t j qtdt|dt|dddt d|||d<|S)NrrrtDOIzFound the DOI(z) for:rpatentthesiszNo authorlist found for  ()zFound the correct DOI:z for the old one:z**z new DOIs has been found and z& incorrect DOIs have been corrected**.redrz4We also have created {} formated author list entries authors_new)rlocrrgrnr[rrrZrbrirrrget_authors_doirr) rZ N_new_doisZN_corrected_doisZ N_authorlistrrrZauthor_Znew_doir5r5r6correct_authors_and_doisJ       (      "rcCsnz8td|ddd}t|jdkr6|jdj}|WSWn YnXt||dd}|dkrd|d StjSdS) z.Looks for the DOI in Scopus and Crossref. ztitle(rT)verboserrN) author_searchrr)Z ScopusSearchraresultsrfind_crossref_for_titlerr)rZZ author_presetZdocument_searchZ doi_scopusrr5r5r6r)s  rcCsd}t}|dkr8|dkr(|j|d}q`|j||d}n(|dkrP|j||d}n|j|||d}|D]B}ztd|dd|}Wnd}YnX|dkrd|SqddS) Ng) bibliographic)rr`)rcontainer_title)rr`rrZrg?)rqueryrratio)rZrrsimrZw1rZsim_newr5r5r6r?s"  rc Cs<z t|WSzt|WYSYYdSXYnXdSN)rr)rr5r5r6r\s rc Csv|dkrt}||}d}dd|dD}t|D],\}}|t|ddkr\|d7}|dd krt|d7}|d D]}|d d kr|d d krt|d dkr|d D]}||d d7}q|ddd}n||d d7}q|d |dd7}q|dd kr0|d}q6|drT||dd7}q6||dd7}q6|dd}|S)z For a given DOI the authors will be searched in Crossref and will be formated in the following way: 'E. Krok, S. Balakin, J. Jung, F. Gross, J. Opitz, and G. Cuniberti'' NrtcSsg|]}d|kr|qS)familyr5).0xr5r5r6 rsz+format_authors_crossref..r`rand rr:givenrr–.-r~rr Cuniberti, , r)rr enumeratera capitalizerpupperisupper) rrrrrrpersonfirstspr5r5r6rhs2   rcCs||dkrt|}d}t|jD]J\}}|t|jdkrB|d7}|jdkrX|d7}|jdkr|jdkr|jD]}|ddkr|dd krt|ddkr|dD]}|t|d d 7}q|dd d }n|t|d d 7}qz|dt|d d 7}qz|jdkr4|d}q|j rX|t|jd7}q|t|jd7}q|dd}|S)z For a given DOI the authors will be searched in Scopus and formated in the following way: 'E. Krok, S. Balakin, J. Jung, F. Gross, J. Opitz, and G. Cuniberti'' Nrtrrr:rrrr r r~rrr r r) rrrrasurnamerrrp html_codingrr)rrrrrrrr5r5r6rs0   rcCstt|}|S)z&Converts string to HTML entities. )neZnamed_entitiesrescape)rr5r5r6rsrcOCsd}dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddN}t|d}|jD]}|D]\}}||jkrt|j||s|j||dkst|j||dkrnj|d krdt|j|d sd| ||t|j|d d n$| ||t|j||d qq| |qW5QRXt d |d S)z Writes the Dataframe df in the structure needed by the webpage to be imported. The database is saved in the file 'filename'. rrrr)NAGr?r firstlastcoverr_rZrpubdate referencerrfilenamenonmieiECEMPCover publisher submisdateISBNbbllxbbllybburxbburyfieldhyperrefr journalrefarxiv arxivmeshendpage refreport_no report_no startpagerjournalacronymWCUISSNrrIF acceptdate revisedate webpubdateESFZIHpudate booktitleeditorsorder_itpubtownreference_morepISSNaddress WCUTOPTEN stampamelovjntref openaccessAG2 journalcover arxivdate commentcv addmaterial receivdateIFyearPRaccessioncode PRstatusURLIFrefplacemisurelibrarycongressseries coverdetailsreferenceshortIEEEcnINSPECan econophysicsother_order_it senzaISBNlinkr>rrtrrrzDatabase written to file N) r'rrcolumnsrgrnrr[rr(rb)rrrseprrkeyspacingr5r5r6write_corrected_dbs   F && r`FcCs|dkrttdtdt|}t|}|jdgd}t}|jD]}|j |j ddd}qL|j|d<|j d ddd ||d  }|d  d |d <|S) z Receives all publications from Google Scholar for a given searchfield Returns ------- DataFrame DataFrame with all publications on Google Scholar with short info (title, year, bib) TzOIf nothing happen in the next minute your IP might already be blocked by GooglezTry: proxy, VPN, TOR, ...rT)sectionsFrVrYr_) ascendingrrZr[)use_proxy_scholarlyrbrcrdrfrergrhrTrirUZ sort_valuesrjrk)Z searchfield use_proxyrqr`rrrMr5r5r6get_pub_googles    recCsDtdtdddddgd}tj||dtd t|dS) NzLet's try to use a proxy...TrDEUSCA)randtimeout country_id)httphttpsz(**proxy is used to access Google with:**)rbrrrcrdrr[)proxyr5r5r6rcs rccCs|tdddt}|jD]>}t|dj||j|d|jddd}t d q|d  d |d <td dd|S) NzCFill all informations available at **Google** for the publications:bluerrZrYTFrVr_roz **Finished**) rrgrhrrbrrirerUtimesleeprk)ZDF pub_filledrr5r5r6google_details_publication s    rtr:c Csggggggggf\}}}}}}}} |d} |d} |d} td|D]\} }t|dt|d||d}|dkrz>t|dd d }z||jWn|tjYnXWn,t |dd d}|tjYnX||d||dd |||t || t |z0|j dkrZ||j n||d d Wn<z||d d Wn|tjYnXYnX| | dkrHzB|j dkrt|j |j| df<n|ddd |j| df<WnJz |ddd |j| df<Wntj|j| df<YnXYnX| | dkrzt|d|j| df<Wn`z6|jdkrtj|j| df<nt|j|j| df<Wntj|j| df<YnXYnX| | dkrXz|d|j| df<Wn`z6|jdkrtj|j| df<nt|j|j| df<Wntj|j| df<YnXYnXd|dddksd|dddkrt||}|dkr||dn(t|dkr||n||dn||dqX|d||d|d||d|d||d|tj| tjqXt||d<t||d<t||d<t||d<t||d<t||d<t||d<|dt|d<t| |d<t|}|S)z Function that gets DOI and title from Crossref and/or Scopus and writes it into the dataframe. If volume or issue is not given by Google it will search for it at Crossref and Scopus. rrr3z%searching in crossref and scopus for:rZrNrFULL)viewz8**not found in scopus**. API key and permission correct?rrTrrru…rz Full-Text new_titlerrr cover_date first_last)rjrbrrrrirrrrrfirst_last_authorrrr[rrprZissueIdentifierrraarrayrkclean_abstracts)rsr`Zdoi_listZ title_list item_listrrZ abstract_listrzr{Zvolume_conditionZnumber_conditionZpages_conditionelementrrrrr5r5r6get_details_crossref_scopuss             ,       rcCsd}z|ddd}|dd}|dd}|dd}t|dkr|dkrT|WS|jdkrv|jdkrn|WS|j}n |j}|WS|WS|dkr|YS|jdkr|jdkr|YS|j}n|j}YnX|S) z Receives abstracts for publication from Crossref and Scopus, given the item found by Scopus and Crossref. Returns ------- String abstract Nrz rtzz z rx)rra descriptionr)rrrr5r5r6rs2          rcCs"|jD]}d}|j|j}|ddkshd|ddkshd|ddkshd|ddkshd |dd kr|d d kr|||d d dd}nD|dd kr|||ddd}n|||ddd}n2d|ddkr||d|d}n||}||j|df<q|S)Nrtrrurvrwrxryrzr{r|r}r~rrrrrrrr)rrrr)rrrrr5r5r6r~s  L r~add.database.paperscPCs&g}|jtjddd}d}ddddd d d d d ddddddg}ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddO}t|d0}|D]\}}z2t|ds*|ddkr*|d} ntWn6zt|d} Wnt |d} YnXYnXt |} t |d} t |d} d| d d!kr| |d"| |d#d$d%|d&d'd(t|d)| | |d*| |d+|d"| d'g} t|| D]\}}| ||d,qqW5QRX|S)-a= Write publications from Google to filenname with the structure needed by the webpage of the chair. It aslo returns the urls of the articles with incomplete abstracts. 'XXX' is written in the database where no data can be found. By searching for 'XXX' later we can complete those entries by hand. ZXXXT)regexrzAG z cvselected ztype z firstlast z #filename z#cover zyear ztitle z authors z pubdate z reference zDOI z hyperref z abstract rrrrz )Orr?rrrr_rZrrrrrrrr r!r"r#r$r%r&r'r(r)r*rr+r,r-rr.r/r0r1rr2r3r4rrr5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r>rrr`ryrz…iNurlZGCnoZtypeXXXr{rtz../pubs/reprints/covers/XXXr_rrr)rrnanr'rrgrn ValueErrorrformat_authors_googleformat_referencerrir(r[rm)rTrZurl_abstract_incompleterZname_rowr]rrrZ author_shortrryrrCnamerr5r5r6add_pubs_to_files       "rc Csd}|dt|dd7}z |dtt|dd7}Wn$|dt|dd7}YnXz0|tt|dd tt|d d 7}Wn4|t|d d tt|d d 7}YnX|S) Nrtzrzz rz r3rr_rr)rr[ro)rrr5r5r6rs 0.rcCs4|ddddks(|ddddkr,dSdSdS) z8 Checks if Cuniberti is either first or last author r`rrr:r~yesrNr5rr5r5r6r| s(r|cCs4t|D]&\}}|dkr$t|qt|qdS)NrT)r webbrowserr'Z open_new_tab)urlsrrr5r5r6 open_urlss   rc Csz|ddd\}}}Wn>z|ddd\}}}WntjYYSXYnXtt|dkrvdt|}tt|dkrdt|}d|||S)Nzpublished-onlinerrZissuedr0z {0}.{1}.{2})rrrar[r)rr_monthdayr5r5r6rs  rc Cs|dkrdd}d}|}|d}t|D]\}}|dd}|t|dkrX|d}|d krh|d }|ddd}d} |D]J} | d d kr| d d kr| || d d7} q| d || dd7} q|| ||}|d kr|}|d}q*|dd}|S)NFcSs|Srr5)rr5r5r6r/sz*format_authors_google..html_codingrtz and rr~rr:zrrr rr r)rprra) autZ html_encoderrrrrrZ first_namesZ first_shortrr5r5r6r-s.   rc Csd}dddddddd g}d d d d d d d d g}tdd}|jD]}|j|ddksf|j|ddkr>t|D]X\}}t|j||s|j||dkrqn||||t|j||dqn||q>q>W5QRXdS)Nrrrr_rZrrrrrrz rZ database_newr>regularletterrr)r'rrrrgrnr(r[)rrrCdistrrrcontr5r5r6write_corrected_database_shortLs  $&*  r)N)N)N)N)F)r:)r)F)BZpandasrgnumpyrrqr,Z namedentitiesrrrZIPython.displayrrosenvironZpybliometrics.scopusrZfp.fprZcrossref.restfulrdifflibrdjango.template.loaderrr django.templater django.httpr r7rDrIrNrRrSrsrrrrrrrrrrrrrrrr`rercrtrrr~rrr|rrrrr5r5r5r6sj       & / M -  % #.   l) >