U
fa· @ sb d dl Zd dlZd dlZd dlZd dlZd dlZd dl m
Z
m Z d dl m a d dl
mZ d dlmZ d dlmZ d dlmZ dd Zd
d Zd d
Zdd Zdd ZdAddZdd ZdBddZ dd Z dd Z d d Z d d Z d!d" Z!d#d$ Z"dCd&d'Z#d(d) Z$d*d+ Z%dDd,d-Z&d.d/ Z'd0d1 Z(dEd3d4Z)d5d6 Z*d7d8 Z+d9d: Z,d;d< Z-d=d> Z.d?d@ Z/dS )F N)Markdowndisplay) scholarly)AbstractRetrieval) FreeProxy)Works)SequenceMatcherc C s t | d}| }W 5 Q R X |d}|D ]D}t|}|i kr,z|j|dd}W q, tj|dgd}Y q,X q,|tj }||d
j j }|j
|dd |S ) NrenditemT ignore_indexr indexZspeakerinplaceopenreadsplit
entry_to_dictappendpd DataFramefillnanpNaNisnar valuesdrop file_namefilecontentZ all_entriesentrydictiodfind r' 0/var/www/nanotud/old_database/modules_webpage.py
read_seminars s
r) c
C s t | d}| }W 5 Q R X |ddd }|D ]t}|d}i }|D ]"}|dd}z|d d kr`|d
kr`d |d d kr`d
|d d kr|d
d |d
< nd|kr|jdddd |d< nd|kr|jdddd |d< nd|jddd}t|dkr&|jddd}t|dkrHtj||d < n|d ||d < W qL Y qLX qL|i kr4z| j|dd} W q4 t j
|d gd} Y q4X q4| tj} | S )u
Reads publication from database given by the format of the webpage.
Parameters
----------
file_name : name of the database file
Structure: Two columns (1 = property, 2 = value )
Entries are separated by 'enditem
##################################'
Example:
'##########################################################################
AG GC
cvselected no
firstlast yes
#filename
cover ../pubs/reprints/covers/2020_Krok.jpg
year 2020
title Modification of titanium implants using biofunctional nanodiamonds for enhanced antimicrobial properties
authors Emilia Krok, Sascha Balakin, Jonas Jung, Frank Gross, Jörg Opitz, G. Cuniberti
pubdate 2020.03.02
type regular
reference Nanotechnology 31, 205603 (2020)
DOI 10.1088/1361-6528/ab6d9b/
abstract The present study describes a novel antimicrobial surface using anodic oxidation of titanium and biofunctional
detonation nanodiamonds (ND). ND have been loaded with antibiotics (amoxicillin or ampicillin) using poly
(diallyldimethylammonium chloride)(PDDA). Successful conjugation with PDDA was determined by dynamic light scattering, …
enditem
###########################################################################
...'
Returns
-------
DataFrame
Containing all publication entries from 'file_name'. Each row contains one publication and
columns its values. If one property is not set in file_name, its value is set to 'np.NaN'.
r r
# r abstract journalfullname maxsplit arxivpasswdTr r
) r r r replacestriplenr r r r r r )
r r! r" Zall_publicationsarticle
attributesr$ attr r% r' r' r( read_publications$ s> $
* r@ c C s | d}i }|D ]}|dd}z|d dkr|dkrd|dd kr|j dd d
}t|d kr|j d d d
}t|d krtj||d < q|d ||d < n|d ||d < W q Y qX q|S ) Nr, r- r. r/ r0 r1 r r* r7 r6 )r r: r< r r r; )r# r> r$ r? r r' r' r( r q s
$ r c C s t | d}| }W 5 Q R X |d}|D ]D}t|}|i kr,z|j|dd}W q, tj|dgd}Y q,X q,|tj }||d
|d
@ j j }|j
|dd |S )
Nr r
Tr r r
ZvornameZnachnamer r r r' r' r( read_addresses s
rB c C s d}t | d}||jdddd | D ]t\}}|| }| D ]J\}}t|dkr~||d t| d qN||d t| d qN|| q0W 5 Q R X d S ) N:enditem
#################################################
wr, r* r7 r. )r writer iterrowsr itemsr< str) r r% seperatorr! r rowto_writeZindex_wvaluer' r' r( write_addresses s rO c C s d || }tt| d S )Nz {})formatr r )stringcolorZcolorstrr' r' r( printmd s rS c C s d}d}d}g }| j D ]}| j| j}t|s@t|d}t|sR|dkrt| j| j}|| j|df< t|st dt| d | j| j |d7 }t|r|
t j | j| j
dkr| j| j
d krt d
| j| j d | j| j
d qt|}|d kr$|
| |d7 }qt| j| j}t|sJ||krX|
t j qt d
| d| |d7 }|| j|df< t|}|d kr|
| |d7 }q|
t j qtdt| dt| ddd t d| || d< | S )Nr r6 r1 DOIzFound the DOI(z) for:r* ZpatentZthesiszNo authorlist found for ()zFound the correct DOI:z for the old one:z**z new DOIs has been found and z& incorrect DOIs have been corrected**.redrR z4We also have created {} formated author list entries authors_new)r locrT r r rJ r; find_doi_for_titletitleprintr r r typeget_authors_doirS rP ) r% Z
N_new_doisZN_corrected_doisZ N_authorlistrY r& rT Zauthor_Znew_doir' r' r( correct_authors_and_doi sJ
(
"r` Cunibertic C sn z8t d| ddd}t|jdkr6|jd j}|W S W n Y nX t| |dd}|dkrd|d S tjS dS )
z.Looks for the DOI in Scopus and Crossref.
ztitle(rV T)verboser* r N)
author_searchjournalrT )Z ScopusSearchr< resultsdoifind_crossref_for_titler r )r\ Z
author_presetZdocument_searchZ
doi_scopusitemr' r' r( r[ s
r[ c C sh d}t }|d kr"|j| |d}n|j| ||d}|D ],}td |d d | }|dkr6| S q6d S )Ng )
bibliographicauthor)ri rj Zcontainer_titler\ r g?)r queryr ratio)r\ rc rd simworksZw1rh Zsim_newr' r' r( rg s
rg c C s< z
t | W S zt| W Y S Y Y d S X Y nX d S )N)format_authors_scopusformat_authors_crossref)rT r' r' r( r_ s
r_ c C s t }|| }d}dd |d D }t|D ]@\}}|t|d d krT|d7 }|d dkrl|d 7 }|d
D ]}|d d kr|d d
krt|d dkr|d D ] }|t|d d 7 }q|dd d }n|t|d d 7 }qx|d t|d d 7 }qx|d dkr4|d }q.|d r\|t|d d 7 }q.|t|d d 7 }q.|dd }|S )z
For a given DOI the authors will be searched in Crossref and will be
formated in the following way:
'E. Krok, S. Balakin, J. Jung, F. Gross, J. Opitz, and G. Cuniberti''
r1 c S s g | ]}d |kr|qS )familyr' ).0xr' r' r(
s z+format_authors_crossref..rj r* and rq ra givenr - –.-Nr/ r6 . Cuniberti, , r+ ) r rf enumerater<
capitalizer html_codingupperisupper) rT rn rh rQ author_listipersonfirstspr' r' r( rp s0
rp c C st t | }d}t|jD ]J\}}|t|jd kr:|d7 }|j dkrP|d7 }|jdkr|jdkr|j D ]}|d d kr|d d
krt|d dkr|d D ] }|t|d d 7 }q|dd d
}n|t|d d 7 }qr|d t|d d 7 }qr|j dkr,|d }q|j
rP|t|j d 7 }q|t|jd 7 }q|dd }|S )z
For a given DOI the authors will be searched in Scopus and formated in the following way:
'E. Krok, S. Balakin, J. Jung, F. Gross, J. Opitz, and G. Cuniberti''
r1 r* ru ra rv NNoner rx ry rz r/ r6 r{ r| r} r+ ) r r~ authorsr< surnamer Z
given_namer r r r )rT r= rQ r r r r r' r' r( ro 3 s.
ro c C s t t| } | S )z&Converts string to HTML entities.
)nenamed_entitieshtmlescape)rQ r' r' r( r U s r c O C s d}dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddN}t |d}| jD ]}| D ]\}}|| jkrt| j| | s| j| | dkst| j| | dkr nj|d krdt| j| d
sd| || t| j| d
d n$| || t| j| | d qq| | qW 5 Q R X t
d | d
S )z
Writes the Dataframe df in the structure needed by the webpage to be imported.
The database is saved in the file 'filename'.
rC rF r. r6 )NAG
cvselectedr^ firstlastcoveryearr\ r pubdate referencerT r3 filenamenonmieiECEMPCover publisher
submisdateISBNbbllxbbllybburxbburyfieldhyperrefr5
journalrefarxiv arxivmeshendpage refreport_no report_no startpagevolumejournalacronymWCUISSNissuepageIF
acceptdate
revisedate
webpubdateESFZIHpudate booktitleeditorsorder_itpubtownreference_morepISSNaddress WCUTOPTEN
stampamelovjntref
openaccessAG2 journalcover arxivdate commentcv addmaterial
receivdateIFyearPRaccessioncode PRstatusURLIFrefplacemisurelibrarycongressseries coverdetailsreferenceshortIEEEcnINSPECan econophysicsother_order_it
senzaISBNlinkrD r r1 r rY r, zDatabase written to file N) r r rI columnsr r rZ rJ r; rG r] )r% r rK sepr! r& keyspacingr' r' r( write_corrected_db\ s
F &&
r Fc C s |dkrt td td t| }t|}|jdgd}t }|jD ]}|j |j
ddd}qL|j|d< |j d ddd
||d }|d
d |d < |S )
z
Receives all publications from Google Scholar for a given searchfield
Returns
-------
DataFrame
DataFrame with all publications on Google Scholar with short info (title, year, bib)
TzOIf nothing happen in the next minute your IP might already be blocked by GooglezTry: proxy, VPN, TOR, ... publications)sectionsFr sort
google_pubr ) ascendingr r\ rJ )use_proxy_scholarlyr] r Z
search_authornextfillr r r r bib sort_valuesisnullastype)Z searchfield use_proxyZ search_queryrj Z
pub_googlepubr' r' r( get_pub_google s
r c C sD t d tdddddgd } tj| | d td t| d S )
NzLet's try to use a proxy...Tr* ZDEZUSCA)randtimeoutZ
country_id)httphttpsz(**proxy is used to access Google with:**)r] r getr r rS rJ )proxyr' r' r( r s
r c C s| t ddd t }| jD ]>}t| d j| |j| d | jddd}t
d q|d
d |d
< t d dd |S )
NzCFill all informations available at **Google** for the publications:bluerX r\ r TFr r intz **Finished**) rS r r r r] rZ r r r timesleepr )ZDF
pub_filledr r' r' r( google_details_publication s
r c C s g g g g g g g g f\}}}}}}}} | d }
| d } | d } td | D ]\}
}t|d t|d ||d }|dkrz>t|d d d
}z||j W n |tj Y nX W n, t |d d d}|tj Y nX ||d ||d d || |t
| | t | z0|j dkrZ||j n||d
d W n< z||d
d W n |tj Y nX Y nX | |
dkrHzB|j
dkrt|j
| j|
df< n |d dd | j|
df< W nJ z |d dd | j|
df< W n tj| j|
df< Y nX Y nX |
|
dkrzt|d | j|
df< W n` z6|jdkrtj| j|
df< nt|j| j|
df< W n tj| j|
df< Y nX Y nX | |
dkrXz|d | j|
df< W n` z6|jdkr tj| j|
df< nt|j| j|
df< W n tj| j|
df< Y nX Y nX d|d dd ksd|d dd krt||}|dkr||d n(t|dkr|| n||d n||d qX|d ||d |d ||d |d ||d |tj | tj qXt|| d< t|| d< t|| d< t|| d< t|| d< t|| d< t|| d< | d t| d< t| | d < t| } | S ) z
Function that gets DOI and title from Crossref and/or Scopus and writes it into the dataframe.
If volume or issue is not given by Google it will search for it at Crossref and Scopus.
r numberpagesz%searching in crossref and scopus for:r\ rd NrT ZFULL)viewz8**not found in scopus**. API key and permission correct?r zshort-container-titleTr rx r u …r3 z Full-Text new_titlerh pub_date
journal_short
cover_date
first_last)r r] rH rg r r Z coverDater r rS pub_date_crossfirst_last_authorZsourcetitle_abbreviationZ startingPagerJ rZ r r ZissueIdentifier get_abstractr< arrayr clean_abstracts)r rj Zdoi_listZ
title_listZ item_listr r Z
abstract_listr r Zvolume_conditionZnumber_conditionZpages_conditionelementrL item_CR item_scopusr3 r' r' r( get_details_crossref_scopus s
,
r c C s d}z| d dd}| dd}| dd}| dd}t|dk r|dkrT|W S |jdkrv|jdkrn|W S |j}n |j}|W S |W S |dkr| Y S |jdkr|jdkr| Y S |j}n|j}Y nX |S ) z
Receives abstracts for publication from Crossref and Scopus, given the item found by Scopus and Crossref.
Returns
-------
String
abstract
Nr3 z r1 zz
z r )r: r< descriptionr3 )r
r r3 r' r' r( r ( s2
r c C s" | j D ]}d}| j| j}|d dkshd|dd kshd|d d kshd|dd kshd |d d
kr|d d kr|||d d
d d }nD|dd kr|||dd d }n|||dd d }n2d|dd kr||d |d }n|| }|| j|df< q| S )Nr1 r ©Z Copyright u c○ Z COPYRIGHTzThis journal is ZWeinheimr/ r, Z Switzerland .r* ir3 )r rZ r3 find)r% numrQ txtr' r' r( r Q s L r add.database.papersc P C s& g }| j tjddd} d}ddddd d
d d d
ddddddg}ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddO}t|d0}| D ] \}}z2t|d s*|d dkr*|d } nt W n6 zt|d } W n t |d } Y nX Y nX t
|}
t |d } t |d } d | d d! kr| |d" |
| d#d$d%|d& d'd(t|d) | | |d* |
|d+ |d" | d'g}
t||
D ] \}}|
|| d, qqW 5 Q R X |S )-a=
Write publications from Google to filenname with the structure needed by the webpage of the chair.
It aslo returns the urls of the articles with incomplete abstracts.
'XXX' is written in the database where no data can be found. By searching for 'XXX' later we can complete
those entries by hand.
XXXT)regexrC zAG z cvselected ztype z
firstlast z
#filename z#cover zyear ztitle z authors z pubdate z
reference zDOI z hyperref z abstract r
rF r. r6 z )Or r r^ r r r r\ r r r rT r3 r r r r r r r r r r r r r r5 r r r r9 r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r rD rY rh rj r r3 z…iNurlZGCnoZtypeXXXr r1 z../pubs/reprints/covers/XXXr r rT r, )r: r nanr rH r r
ValueErrorrp format_authors_googleformat_referencer r rG rJ zip)r r Zurl_abstract_incompleterK Zname_rowr r! r rL Z author_shortr r r3 r" namerN r' r' r( add_pubs_to_filee s
"r# c C s d}|dt | d d 7 }z |dtt| d d 7 }W n$ |dt| d d 7 }Y nX z0|tt| d d tt| d
d 7 }W n4 |t| d d tt| d
d 7 }Y nX |S )
Nr1 zr zz r z r rU r rV r )r rJ r )rL rQ r' r' r( r s 0.r c C s4 | d d d dks(| d d d dkr,dS dS dS ) z8 Checks if Cuniberti is either first or last author
rj r rq ra r/ yesr Nr' )rh r' r' r( r s (r c C s4 t | D ]&\}}|dkr$t| qt| qdS )Nr T)r~
webbrowserr Z open_new_tab)Zurlsr r r' r' r( open_urls s
r& c C s z| d d d \}}}W n> z| d d d \}}}W n t j Y Y S X Y nX tt|dkrvdt| }tt|dkrdt| }d|||S )Nzpublished-onlinez
date-partsr Zissuedr* 0z {0}.{1}.{2})r r r< rJ rP )rh r monthdayr' r' r( r s r c C s d}| }| d}t|D ]\}}| dd }|t|d krH|d }|dkrX|d }| dd d }d}|D ]J}|d d kr|d d
kr|t|d d 7 }qr|d t|d d 7 }qr|| t| }|dkr|d }|d
}q|d d }|S )Nr1 z and r6 r/ r* ra rv r rx ry r{ zr} r+ )r r~ r< r ) ZautrQ r r r" r Z first_namesZ first_shortr r' r' r( r s*
r c C s d}dddddddd g}d
d d
d
d
d d d
g}t dd}| jD ]}| j| d dksf| j| d dkr>t|D ]X\}}t| j| | s| j| | dkrqn|||| t| j| | d qn|| q>q>W 5 Q R X d S )NrC r^ r r r\ r rT r r3 rF r. z r6 Z database_newrD regularletterr r, )r r rZ r~ r r rG rJ )r% rK r" distr! r& r contr' r' r( write_corrected_database_short s
$&* r. )N)ra )F)ra )r )0pandasr numpyr r
namedentitiesr r r% IPython.displayr r r Zpybliometrics.scopusr Zfp.fpr Zcrossref.restfulr difflibr r) r@ r rB rO rS r` r[ rg r_ rp ro r r r r r r r r r# r r r& r r r. r' r' r' r( sL M
-
$".
l)
>