nanoHUB Citations in the Scientific Literature
nanoHUB citations in nano-Research, Education, and Cyberinfrastructure
We have identified 575 citations to nanoHUB in the scientific literature with 56% of these citations attributed to investigators who are not in any way affiliated with NCN. The citation map provides a visual representation of affiliated and non-affiliated investigators. Non-affiliates appear outside of the dashed line. Researchers affiliated with but not funded by NCN are responsible for many of the remaining citations. NCN clearly is strongly networked through research papers, with networks being developed outside of NCN. Some of the outside networks are completely decoupled from NCN. 509 (89%) of the citations appear in peer-reviewed journals, conference proceedings, book chapters. Ph.D. or Master’s thesis. 469 (82%) pertain to research in the nanotechnology field. Thus, the vast majority of citations refer to actual nanoHUB.org use.
Two years ago we began charting citation network maps to address the question of whether nanoHUB can indeed be used for research. The documented citations and their extent into the non-NCN-affiliated nano community exceed those of any other science gateway we are aware of. Next the question arose: “Is it good research?” In the past year we have begun to address that question by asking: “Are the papers that cite nanoHUB subsequently cited by other authors?” We have interacted with Luo Si, professor of computer science at Purdue, who helped us to mine the Google Scholar service to obtain the secondary citations to the nanoHUB citations. Let’s imagine nanoHUB as the author of the 575 papers citing nanoHUB. We have found 3,251 citations to these 575 primary papers such that the h-index is 27. That means 27 of the primary papers have at least 27 citations. Considering that the first primary papers appeared in 2000, the “beginning of nanaoHUB’s scientific career,” nanoHUB exceeds the typical value of 10 for the h-index of a professional with 10 years of experience.
nanoHUB use by Experimentalists
Documenting the impact achieved by researchers who are using a remote cyberinfrastructure is a challenging task and part of our assessment effort. This year we have re- examined all 575 nanoHUB citations in the literature to find out if the citation is either given by experimental group that has clearly designed or improved an experiment and utilized nanoHUB resources along the way, or, which is a bit easier to identify, if the paper is plotting real experimental data. In the cosmos of 469 citations that reference nanoHUB usage in nano research, we have identified 55 (12%) papers that are clearly driven by experimentalists, and 142 (30.3%) papers that plot experimental data. We consider these numbers to be a strong evidence of extensive use of nanoHUB by experimentalists doing experimental nanotechnology.
We have also determined that 41 of the papers are written by authors with industrial affiliations. This is just about 9% of the 469 nano research papers. As a concrete example we reference an experimental Phys. Rev. B paper where the authors use the conceptual model of molecular conduction in the tool MOLCtoy by Supriyo Datta to explain fundamental conduction results.
The authors state in the abstract: “A fewparameter scalar model for ballistic current flow through a single energy level is sufficient to describe the main features observed in scanning tunneling spectra of individual Mn12 molecules and offers a deeper insight into the electronic transport properties of this class of single-molecule magnets.” The authors plot their data and MolCtoy results in their paper on the same chart as shown in the figure here. The authors logged into nanoHUB some 65 times and ran around 280 simulations with MolCtoy consuming less than 20 minutes of CPU time. This paper was published in Oct. 2008 and already has been cited three times.
As another example we list the use by Judy Hoyt’s MIT research group of nanoHUB FETtoy tool to examine their experimental Si/Ge slabs. The work is published in IEEE transactions electron devices. Prof. Hoyt, who is a very well respected experimentalist in the nanoelectronic community and a Fellow of the IEEE, is planning to attend the NCN site visit virtually through a teleconference link to describe her nanoHUB interactions.
A newly developed nanoHUB tool that modelselectron transport in high mobility InAs/InGaAs based transistors. The particularly important modeling capability is the mapping of the non-parabolic bandstructure to a simplified model and the ability to compute the gate tunneling. The visualization of the current flow drove a nanoHUB development that now enables an intuitive representation of gate tunneling. The new OMENfet code has now been released and is now available openly on the nanoHUB. The scientific results generated in this experimental and theoretical collaboration has been published in a co-authored IEDM proceedings article involving the experimental MIT group of Jesus del Alamo and the theory group of Klimeck at Purdue. The funding for the science was obtained through leveraged grants of the FCRP / MSD center where del Alamo and Klimeck participate.
The analysis of our external literature citations revealed some examples where external theory-driven researchers utilized a nanoHUB tool that has been contributed to nanoHUB from outside of the NCN. For example the authors Kureshi and Hasan working at a university in India published their study CNT bundles as possible copper replacement interconnects in the “Journal of Nanomaterials” in April 2009. In their work they utilized the tool “Carbon Nanotubes Interconnect Analyzer (CNIA)” for over 1,600 simulations in the year 2009 alone. CNIA was contributed by Tanachutiwat and Wang of the University of Albany. We cannot see any relationship between the two research groups and NCN other than the contributed, efficiently served, and utilized tool CNIA. The same CNIA tool has previously also been cited in a Master’s thesis at the University of Cincinnati in 27 the year 2005. The concept of community contributions coming from outside the NCN and being useful for an unrelated research group is becoming reality.
Lowering Barriers to Simulation
Accelerating Deployment of Scientific Software
In June 2005, NCN released Rappture 1.0, a new, open toolkit to facilitate the rapid development and robust deployment of scientific simulation packages. Designed to work as a library for a variety of programming languages, including Matlab, C/C++, Fortran, Python, Perl, Tcl, and Ruby, Rappture automatically generates a graphical user interface from a description of the inputs and outputs of the simulation program. Rappture has become the main vehicle for software deployment on nanoHUB. Students across the NCN network are using Rappture to enable simulation applications for use on nanoHUB.org. Simulation usage on nanoHUB has grown seven-fold to over 7,900 users in less than four years and we attribute this increase to the friendly interfaces afforded by Rappture.
Prior to Rappture, a significant portion of nanoHUB users downloaded software and installed it on their own computers. However, since the introduction of Rappturized applications, the number of software downloads has all but vanished – Rappturized software is user-friendly. Rappture is also central to NCN’s long-term strategy to link simulations in complex workflows. Because of the scale of the NCN initiative, we expect that Rappture will have broad impact on scientific computing outside NCN. There are currently over 282 Rappture software development projects under way with 318 developers. In all, 140 tools have been deployed on nanoHUB, with all but seven using Rappture. Although we do not force developers to use Rappture, approximately 95% of the deployed projects use it, which indicates developer buy-in and the ease of development with Rappture technology.
Leadership in Assessment and Virtual Organizations
Open Usage Statistics
The idea of science portals that enable the rapid dissemination of scientific and engineering results (and that enable other researchers and educators to use these results) has been pursued by many organizations since the early-to-mid-1990s. The Purdue University Network Computing HUB (PUNCH) was one of the first. We believe there are five critical elements of a successful science gateway:
1) Connection to outstanding science/engineering. 2) Willingness to make the results useful to others, outside the core community. 3) Efficient, dependable infrastructure operations. 4) Technology that enables rapid development and deployment. 5) Open assessment and usage statistics.
We believe that while most science gateways are based on criterion 1), most struggle to meet points 2) and 3) and are lacking 4) and 5). HUBzero can help to address 3), 4) and 5) as discussed above. Even extremely well–funded NSFbased infrastructures, such as TeraGrid or NEES, do not make their usage data and usage patterns as available as does nanoHUB. Usage data guides nanoHUB technology development, and its availability to our contributors and users bolsters our engagement efforts. We believe that the detailed study and openness of the 46 nanoHUB usage statistics has advanced nanoHUB capabilities and has given nanoHUB a standing as the premier science gateway. nanoHUB can now provide contributors with usage and impact statistics that can be used in proposals as hard evidence of their impact on a community.
The figure above for example shows the monthly and cumulative numbers of user served by Prof. Dragica Vasileska at Arizona State University.
Content Characterization by Usage, User Feedback, and Community Involvement
As nanoHUB content increases, we find that users are struggling to find the high quality content. As a result, we have continued to improve the search mechanisms on nanoHUB to enable rapid information retrieval. One key element in this effort is to characterize each content item by a variety of criteria that ultimately influence the ranking of the resource. Each simulation tool is characterized by:
1) A Google-like ranking based on user reviews and use. 2) A target audience rating, or, the expertise level expected from the user. 3) An indication if this is an NCN Supported tool, or a community supported tool 4) Data including number of users and simulation jobs, average run time, and average number of stars awarded in reviews. 5) Number of citations in the scientific literature–this indicates the vetting of the tool and its use in research. 6) Number of questions, indicative of the liveliness of the community. A large number of open questions suggests a poorly supported tool. Conversely, large numbers of closed questions indicates a live code with tool owners interested and dedicated to its support. The introduction of a virtual economy has proved to have a positive influence on the question and answer forum. 7) A wishlist enables users to express tool improvement wishes and the tool development team to handle tool improvement processes. 8) User reviews: anyone can give a 0- to 5-star review and submit written comments. 9) Users can also declare nanoHUB content items as their favorites, which they can later easily find again on their favorite list. Furthermore, they can share their favorite nanoHUB items on six different social network sites, including Facebook, Twitter, and Google. 10) A list of associated and recommended documents that support this tool.
Processes for User Surveys
Dr. Diane Beaudoin, Director of Assessment for the College of Engineering at Purdue, has served as NCN Director of Assessment for two years. She leads the effort to formalize nanoHUB assessment and user survey processes. Last year we had begun a systematic user survey process. We have categorized registered users by their usage patterns:
1) one-time, 2) nonsimulation, and 3) heavy users.
The so-called “One-time” users utilize nanoHUB content for a single visit only and never return. We have also devised specialized user surveys for these groups. One interesting result of the specialized survey that went to the “one-time” users (which was accommodated by a large 10% return of survey requests) is that these one-time users are overall quite satisfied with what they received from nanoHUB. Another surprising result was that users want to interact with other users more. This has driven nanoHUB component developments that enable connections to social network sites.
This reporting year we contacted 1,431 users in December 2009 who had registered their account within the prior 3 months. Also on this survey we received a very high 9% response rate with 130 people. 52% were not using nanoHUB as part of a course. Of these, 49% were graduate students, 21% were professional scientists/engineers, and 16% were faculty members. Users who utilize the nanoHUB primarily in the context of coursework – we call course users. Of these course users, 90% were undergraduate students. The majority of our new non-course users (users of nanoHUB for purposes other than coursework) discovered nanoHUB by surfing the web. This result reconfirmed our effort that we must improve our presence and linkage to other web sites like Wikipedia and iTunes U, which helps our Google ranking.
Components of these surveys address research questions that the Education Research team has posed. Finally, we are also working with external groups that have approached the nanoHUB team because of a desire to study nanoHUB as a virtual organization and coordinate the user populations that are being surveyed.
How users found out about nanoHUB
Independent Studies by VOSS Projects
In the spring of 2008, NSF solicited proposals for studies on “Virtual Organizations as Sociotechnical Systems (VOSS)”. A portion of the program announcement reads as follows: “A virtual organization is a group of individuals whose members and resources may be dispersed geographically, but who function as a coherent unit through the use of cyberinfrastructure. Virtual organizations are increasingly central to the science and engineering projects funded by the National Science Foundation. Focused investments in sociotechnical analyses of virtual organizations are necessary to harness their full potential and the promise they offer for discovery and learning. The Virtual Organizations as Sociotechnical Systems (VOSS) program supports scientific research directed at advancing the understanding of what constitutes effective virtual organizations and under what conditions virtual organizations can enable and enhance scientific, engineering, and education production and innovation. …..” Two proposal teams, one each from Northwestern (PI, Noshir Contractor) and Purdue (PI, Michael Beyerlein), approached NCN to provide nanoHUB user and usage data and access to nanoHUB users for interviews. Both teams were successful, and are working with each of them. From these collaborations, we expect to gain further insight into our own virtual organization.
VOSS Survey Documents Impact of nanoHUB on Research
The Purdue VOSS team distributed an online survey to 3,940 nanoHUB users who have been active in the last three years. There were 278 respondents to the survey of which 186 completed the full survey. The survey asked users about how nanoHUB has impacted their work and the results is charted below.
Seventy percent of respondents noted that nanoHUB meets their research needs; 50% note that nanoHUB has accelerated their research work. In addition, around 50% say that nanoHUB changes are increasing nanoHUB value to them. The dominantly neutral response on the issues of feedback and response to feedback shows room for improvement.
nanoHUB Virtual Economy Shows Impact
The concept of a virtual economy is most commonly associated with multi-player online gaming communities. nanoHUB.org has borrowed the concept to assist in its own community building, resource management, and sustainability goals. As of March 2008, the system allowed users to earn points by asking and answering questions in the Answers forum. They could spend points on merchandise in the nanoHUB.org store, or spend them by asking questions in the forum and assigning a point reward for the best answer.
In January 2009, an assessment project was conducted as part of a nanoHUB staff member’s Master’s thesis with a goal to evaluate the early impact of the new virtual economy system at nanoHUB.org on user behavior, as well as to establish a generic assessment model for the nanoHUB virtual economy. The study analyzed site usage data for equal time periods before and after the introduction of the first few components of the system. The analysis showed that there was a substantial increase in the number of Q&A contributions per user in the post time period in all categories and groups of users. The maximum (9.6 times) increase was seen in a category of heavy simulation users (users with over 30 and below 350 simulation runs in a 6-month period). Expert users (those with over 350 simulation runs in 6 months) had highest ratios of Q&A contributions per user, going from over 0.22 to almost 0.97 in the post period. Similarly, there was an 8.3 times increase in proportion of contributing users per all users within the category of heavy users.
As a result of the study we have further expanded our reward system to encourage good behavior on the site. Points are now rewarded for meaningful comments and ratings – other users have to give these ratings a “thumbs-up” before any points are rewarded. We also fully introduced point rewards for contributed tools and seminars as they are being used on the nanoHUB.
nanoHUB Community Building by Job Announcements
NCN also created a new area of nanoHUB.org to manage job applications for the nanotechnology community. Any member can post a resume as part of their profile. Professors and other employers can also post specific job advertisements. Students can browse the ads and apply for particular jobs, either directly on nanoHUB.org or off on the employer’s web site.
We created this capability not only to help forge connections within the community, but also to experiment with a new model for sustainability. Many institutions pay significant fees for job advertisements. Posting a single job ad to HPCwire.com, for example, costs $950. Unlike Google AdSense or other advertising programs, having job ads would not hurt the scientific integrity of nanoHUB.org, and would be viewed by users as a service to the community.
As part of the jobs component, we developed an order fulfillment system for subscription services. Right now, we are offering a free “basic” subscription for job postings, which allows one job posting per month. We intend to experiment with other subscription models during the coming year. An advanced subscription would support several job postings at once and unlimited access to our resume database for a monthly fee. We may continue to offer the free subscription as a community service for professors who want to advertise a post-doc position or summer internship at no cost. We plan to expand the use of subscription services to many other areas of the nanoHUB.org site as well, so users can pay for additional disk space, faster turn-around time for simulations, etc. We will establish ways for users to pay with the points they earn while working on the site, and also with real dollars.