GP/EFSA/AMU/2020/03 - Support for Automating some specific steps of Systematic Review process using Artificial Intelligence

Deadline: Jan 15, 2021  
CALL EXPIRED

1. GRANT OPPORTUNITY AND CONDITIONS2

1.1 LEGAL FRAMEWORK

Article 36 of the Regulation (EC) 178/2002 of the European Parliament and of the Council of 28 January 2002 laying down the general principles and requirements of food law, establishing the European Food Safety Authority and laying down procedures in matters of food safety foresees the possibility to financially support networking of organisations operating in the fields within the EFSA’s mission.

In particular, Article 36 (1) stipulates that the Authority shall promote the European networking of organisations operating in the fields within the Authority`s mission. The aim of such networking is, in particular, to facilitate a scientific cooperation framework, the development and implementation of

joint projects3, the exchange of expertise and best practices in the fields within the Authority`s mission.

On the 19th December 2006 the Management Board, acting on a proposal from the Executive Director, drew up a list of competent organisations designated by the Member States which may assist EFSA, either individually or in networks, with its mission. This list is regularly updated by EFSA’s Management Board.

Article 5 of the Commission Regulation (EC) 2230/2004 of 23 December 2004 laying down detailed rules for the implementation of the European Parliament and Council Regulation (EC) 178/2002 with regard to the network of organisations operating in the fields within the EFSA’s mission specifies that the financial support to the networking organisations shall take the form of subsidies (grants) awarded in accordance with the EFSA’s financial regulation and implementing rules.

The present Call for proposals and guide for applicants (hereinafter referred to as “the Call”) is procedurally governed by Title VIII of Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council of 18 July 2018 on the financial rules applicable to the general budget of the Union, amending Regulations (EU) No 1296/2013, (EU) No 1301/2013, (EU) No 1303/2013, (EU) No 1304/2013, (EU) No 1309/2013, (EU) No 1316/2013, (EU) No 223/2014, (EU) No 283/2014, and Decision No 541/2014/EU and repealing Regulation (EU, Euratom) No 966/2012.

This call is based on EFSA’s 2020-2022 draft Work Programme for grants and operational procurements as presented in Annex IX of the draft Programming Document 2020 – 2022, available on the EFSA’s website4.

1.2 BACKGROUND AND MAIN OBJECTIVE OF THE CALL

This call for proposals is divided into two lots. Applicants can apply for one or both lots.

BACKGROUND FOR LOT 1 AND 2

EFSA AMU (Assessment and Methodological Support) Unit is working to automate the process of Systematic Review, one of the pillars of EFSA Scientific Risk Assessment.

A systematic review (SR) aims to provide a complete, exhaustive synthesis of the evidence pertaining to a research question.

Systematic review is a very rigorous process, although expensive and time consuming in terms of human resources. The process is common to EFSA, EMA, ECDC and ECHA, adopted with various forms of implementation. The methodology itself was originally standardised by The Cochrane Collaboration for answering health care research questions (e.g. efficacy of interventions, diagnostic test accuracy questions, questions on prognosis). Nowadays SR has extended to many other research fields and there is a growing number of institutions and research centres working on standards for conducting SR in their field (e.g. EFSA, EBTC, GRACE, Navigation Guide, OHAT/NTP). In addition, different software for supporting the SR process have been developed and implemented (e.g. DistillerSR, HAWC, CADIMA, Covidence, DRAGON, EPPI-Reviewer)

The current methodology is approaching its physical limits as it is barely able to cope with the amount of scientific research available today: such amount is destined to increase dramatically in the future. There are now 75 trials, and 11 systematic reviews of trials, per day, only in the scientific area of Medicine, and a plateau in growth has not yet been reached.

There is as well a growing need to adapt current methodology to a situation where new research appears continuously, i.e., Cochrane’s effort on “Living Reviews” and methods thereof. With a median age of 8 months since last search, most systematic reviews are already outdated on publication.

There are moreover other evolutions proposed as in Cochrane’s specific methods’ groups, including “Automation”, “Rapid Review” and “Living Review”. An essential stakeholder with regard to automation is ICASR (International Collaboration for the Automation of Systematic Reviews), who is working on both methods and tools for implementing automation in SR.

The application of Machine Learning and other Artificial Intelligence techniques has the potential to save costs by reducing the time needed to perform labour-intensive tasks, improve reproducibility, and enable the development, update and delivery of timely and affordable information to decision- makers

MAIN OBJECTIVE OF THE CALL FOR LOT 1

The main objective of lot 1 is to support EFSA in its collaboration with ICASR in order to progress their activities and EFSA vision and strategy on automation of Systematic Reviews.

MAIN OBJECTIVE OF THE CALL FOR LOT 2

The objective of lot 2 is to increase level of automation in data extraction phase, as such area is currently very expensive and error prone. Efforts are already ongoing in EFSA, US EPA (Environmental Protection Agency) and Cochrane, although there still seems to be a long way to go.

Most scientific papers are still published in the form of pdf files and often without standardization. Nevertheless, it is common understanding that the benefits could be very high, both in terms of efficiency and minimization of extraction errors. There is in this area a concrete cooperation with US EPA, as they are already investing in that, at least for some components of the phase, as Named Entity Recognition.

The software tool to be used for the Named Entity Recognition process is being developed by US EPA, which will make it available as open-source and free of charge: the name of the tool is Fiddle and is being developed by Sciome, under a grant with US EPA.

EFSA role in the cooperation is to support in creating training datasets for Fiddle, in the domain area where EFSA has more experience. The activities will therefore consist mainly in two tasks:

1) annotate scientific papers so that they can be used as training databases 2) support in beta-testing the Fiddle tool after that is has been trained

1.3 SPECIFIC OBJECTIVES FOR EACH LOT

LOT 1

The main deliverables will be to advise EFSA on and make suggestions in order to improve its methodology for Systematic Reviews, via proper automation techniques based essentially on Artificial Intelligence and Machine Learning techniques. The content will be derived by closely monitoring activities of ICASR, Cochrane Collaboration and other relevant organizations, and properly supporting EFSA in interacting with them.

Deliverables will be mainly in the format of reports containing advice on how EFSA should evolve its strategy for automating Systematic Reviews. Reports should contain summaries of evolutions appearing in public documentation from above mentioned organizations, augmented with practical and tangible suggestions of how to adapt them to operational reality of EFSA.

LOT 2

The main deliverables should be Training Datasets, including guidance, for Named Entity recognition, to be re-usable either by EFSA alone or in the cooperation with US EPA. In general, the deliverables should not only be annotated documents, rather be annotated documents containing full information about which entities have been annotated. That is important in order to preserve information about the process of identifying entities and how the annotators have come to identify entities, so that the process can be automated using Machine Learning and Artificial Intelligence techniques that have all the information to learn.

Deliverables should include the definition of an Open File Format, so that tools coming earlier and later in the process of Systematic Review, can exchange information with tools dedicated to the Data Extraction phase. EFSA has a similar effort already ongoing in order to define an Open File Format for the Search Strategy phase. The two Open File Formats should be technically compatible with each other and the beneficiary will be informed in due course of the Open File Format being used for Search Strategy by EFSA.

1.4 ELIGIBLE ORGANISATIONS

To be eligible, applicants must be on the list of competent organisations designated by the Member States in accordance with Article 36 of Regulation (EC) 178/2002 and Commission Regulation (EC) 2230/2004. This list is regularly updated by EFSA Management Board and is available for consultation using this link https://efsa.force.com/competentorganisations/s/.

In order to achieve the main objective of the call, proposals can be submitted by one eligible organisation or by a consortium of eligible organisations. In case of a consortium, one of the partners must be identified in the proposal as the consortium leader. The applicant is responsible for identifying consortium partners.

1.5. ROLES AND RESPONSIBILITIES

For proper understanding of this call it is important to have clarity on the terminology regarding involved organisations and their roles.

A) Proposals submitted by consortium:

• The Applicant submits the proposal/grant application to EFSA on behalf of the consortium. The applicant is the leading entity of the consortium. There can be only one applicant in project proposal/grant application;

• The Partner is the other entity in the consortium. There can be a minimum of one partner or more partners.

Once the grant is awarded, the grant agreement is signed between EFSA and the applicant. Partners do not sign the grant agreement directly but instead sign a mandate (template provided by EFSA) authorising the applicant to sign the grant agreement and any future amendments on their behalf.

As soon as the grant agreement is signed, the applicant becomes the Coordinator and partner/s become co-beneficiary/ies. The coordinator and co-beneficiary/ies are referred to as the beneficiaries. The beneficiaries are jointly and severally liable for the technical implementation of the project as described in the proposal which becomes annex 1 of the grant agreement. If a beneficiary fails to implement its part of the project, the other beneficiaries become responsible for implementing that part.

The coordinator has the following important roles:

• Takes part in implementing the project;

• Monitors the action is implemented properly;

• Act as intermediary for communication between the consortium and EFSA;

• Receives and answers all claims EFSA might have in relation to implementation of the project;

• Requests and reviews any documents or information required by EFSA and verifies their

  completeness and correctness before passing them to EFSA;

• Informs EFSA and the partner/s of any event that is likely to substantially affect

  implementation of the project;

• Submits the deliverables and reports to EFSA;

• Requests and receives payments from EFSA and distributes the funds to partner/s without unjustified delays;

The coordinator may not delegate the above-mentioned tasks to the co-beneficiary/ies or subcontract them to any third party.

The other beneficiary/ies:

• Take part in implementing the project;

• Forward to the coordinator the data needed to draw up reports, financial statements and other documents required under the grant agreement;

• Inform the coordinator of any event or circumstances likely to substantially affect or delay the implementation of the project.

B) Proposals submitted by a sole applicant:
• The Applicant submits the proposal/grant application to EFSA. There can be only one applicant in the proposal/grant application.

As soon as the grant agreement is signed, the applicant becomes the beneficiary. The beneficiary is liable for the technical implementation of the project as described in the proposal which becomes annex 1 of the grant agreement.

The beneficiary:

• Takes part in implementing the project;

• Monitors the action is implemented properly;

• Communicates with EFSA;

• Receives and answers all claims EFSA might have in relation to the implementation of the project;

• Requests and reviews any documents or information required by EFSA and verifies their completeness and correctness before passing them to EFSA;

• Informs EFSA of any event that is likely to substantially affect the implementation of the project;

• Submits the deliverables and reports to EFSA;

• Requests and receives payments from EFSA;

1.6. POSSIBILITY OF IMPLEMENTING CONTRACTS AND SUBCONTRACTING

Implementation contracts:

Where the implementation of the project requires the award of procurement contracts (implementation contracts), e.g. purchase of services and/or goods or equipment necessary for the implementation of the action, the beneficiary must award the contract to the entity offering the best value for money or the lowest price (as appropriate), avoiding conflicts of interests. The beneficiary is expected to clearly document the tendering procedure and retain the documentation for the event of an audit.

Entities acting in their capacity as contracting authorities within the meaning of Directive 2014/24/EU5 must comply with the applicable national public procurement rules.

Sub-contracting:

Sub-contractors are not consortium partners and are not party to the grant agreement. They do not have any contractual relationship with EFSA. Subcontractors are entities contracted by the beneficiary to carry out some specific tasks or activities. Subcontracting is allowed under these conditions:

• Subcontracts must be awarded to the entity offering best value for money or the lowest price (as appropriate), avoiding conflicts of interests;

• Subcontracting must only cover the implementation of a limited part of the action;

• Recourse to subcontracting is justified having regard to the nature of the project and what is necessary for its implementation;

• Tasks to be subcontracted and the corresponding estimated costs must identified in the estimated budget and approved by EFSA before the signature of the grant agreement;

• Recourse to subcontracting during project implementation, if not envisaged from the outset

  in the proposal, is subject to prior authorisation in writing by EFSA, and must be formalised via an amendment to the grant agreement. Approval may be granted as long as it does not entail a change to the grant agreement which would call into question the decision awarding the grant or be contrary to the equal treatment of applicants;

• The conditions applicable to the beneficiaries under Articles II.6 (Confidentiality), II.7 (Processing of Personal Data), II.8 (Visibility of Union Funding) of the grant agreement are also applicable to the subcontractor;

• Core tasks must not be subcontracted. Only ancillary and assistance tasks can be subcontracted.

1.7 DURATION, MEETINGS AND REPORTING Lot 1

The maximum duration of the action under lot 1 is 15 months from the kick-off meeting. The following meetings with EFSA are foreseen for lot 1:

1. Kickoffmeeting(tele-meeting):Thekick-offmeetingisregardedasthestartoftheproject and must take place no later than one month after the signature of the grant agreement. At this meeting, details of the project will be discussed and the objectives, the final report structure, deliverables and timeframe will be clarified. In particular, the beneficiary will explain their proposal. Minutes of the meeting shall be taken and provided to EFSA by the beneficiary.

   The presence at the kick-off meeting of a beneficiary’s staff member responsible for administrative/finance issues of the project is advised as this will facilitate understanding by the beneficiary of the grant principles, related financial reporting requirements (declaration and documentation of incurred costs) and significantly ease the financial management of the grant agreement, both for EFSA and the beneficiary.

2. Interimtele-meetingswillbeheldeverythreemonthsfromthekick-offmeeting:The purpose of such meetings is to discuss interim report/deliverable as well as any problems or difficulties (technical or financial) encountered during the project. Minutes of the meeting shall be taken and provided to EFSA by the beneficiary.

3. Final meeting (tele-meeting) will be held one month before the end of the project:

   The purpose of this meeting is to discuss the final report/ deliverables as well as any problems or difficulties (technical or financial) encountered during the project.

The following reports are foreseen for lot 1 and must be drafted in United Kingdom Standard English language and may be subject to publication at EFSA’s discretion.

Please note that all reporting, minutes, outcome of the discussions could be submitted at EFSA’s discretion to EFSA’s Panel and WG members.

Lot 2
The maximum duration of the action under lot 2 is 15 months from the kick-off meeting. The following meetings with EFSA are foreseen for lot 2:

1. Kickoffmeeting(tele-meeting):Thekick-offmeetingisregardedasthestartoftheproject and must take place no later than one month after the signature of the grant agreement. At this meeting, details of the project will be discussed and the objectives, the final report structure, deliverables and timeframe will be clarified. In particular, the beneficiary will explain their proposal. Minutes of the meeting shall be taken and provided to EFSA by the beneficiary.

   The presence at kick-off meeting of a beneficiary’s staff member responsible for administrative/finance issues of the project is advised as this will facilitate understanding by the beneficiary of the grant principles, related financial reporting requirements (declaration and documentation of incurred costs) and significantly ease the financial management of the grant agreement, both for EFSA and the beneficiary.

2. Interim tele-meetings will be held every two months after the kick-off meeting: The purpose of these meetings are to discuss the interim report/deliverable as well as any problems or difficulties (technical or financial) encountered during the project. Minutes of the meetings shall be taken and provided to EFSA by the beneficiary.

3. Final meeting (tele-meeting) will be held one month before the end of the project:

   The purpose of this meeting is to discuss the final report/ deliverables as well as any problems or difficulties (technical or financial) encountered during the project.

The following reports are foreseen for lot 2 and must be drafted in United Kingdom Standard English language and may be subject to publication at EFSA’s discretion.

1. Interimreport:everytwomonthsafterthestartdateoftheproject(thekick-offmeeting) a written interim report must be submitted to EFSA. The written interim report must describe progress and risks in the activity. The report will include presentation of interim results (including graphs, tables, maps and other presentations/visualizations of results). It will as well include detailed analysis on the performance achieved in tracking users’ requests and on users’ satisfaction, related to the activities in scope. With regards the training databases, the frequency for submission to EFSA will be agreed with the beneficiary during the course of project implementation.

2. Final report: 15 months after the start date of the project (the kick-off meeting) a final report must be submitted to EFSA. The final report will comprise the integration of the interim reports and an overall summary, conclusions and discussions for the whole project.

Please note that all reporting, minutes, outcome of the discussions could be submitted at EFSA’s discretion to EFSA’s Panel and WG members.

1.8 PAYMENTS

The following payment scheme will be applied:

• pre-financing payment, upon the entry into force of the grant agreement, without need for payment request, between 10% and 30% of the maximum grant amount; the aim of the pre-financing is to provide the beneficiaries with a float; it remains the property of the EU until the payment of the balance. The exact percentage amount of pre-financing will be determined at the time of grant award;

• final payment (payment of the balance), after the final EFSA grant amount has been determined (in line with Article II.25 of the grant agreement); the amount due as the balance payment is calculated by EFSA by deducting from the final EFSA grant amount the total amount of pre-financing payments already made; if the total amount of earlier payments is greater than the final EFSA grant amount, the payment of the balance takes the form of a recovery; if the total amount of earlier payments is lower than the final EFSA grant amount, EFSA will pay the balance; payment is subject to the approval of the final report by EFSA.

  1.9 GRANT PRINCIPLES

  The financial help provided by EFSA under this Call is a grant governed by the EU Financial Regulation referred to in part 1.1. Accordingly, the grant awarded following this Call must comply with the following principles:

• Co-financing: In accordance with Article 190 of the Financial Regulation, grants shall involve co-financing. The resources necessary to carry out the project /action shall not be provided entirely by the grant. The project costs not covered by the EFSA grant must be financed from the applicant and partner/s resources. The applicant and its partner/s must therefore contribute financially to the project. Additionally, there may be also a financial contribution from another entity, but such an entity may be only a public body. Contributions from the private sector are not permitted.

• No-profit: In accordance with Article 192 of the Financial Regulation, grants shall not have the purpose or effect of producing a profit within the framework of the project for the applicant or partner. Profit is defined as a surplus of the receipts over the eligible costs incurred by the beneficiaries, at the time of request for payment of the balance. The receipts shall be limited to income generated by the project, as well as financial contributions specifically assigned by donors to the financing of the eligible costs. Where a profit is made, EFSA shall be entitled to recover a part of it in line with procedure foreseen in the Grant agreement. The verification of the non-profit rule does not apply to low value grants (</= 60.000 €) or grant in the form of financing not linked to costs pursuant to article 125(1)(a) of the Financial Regulation.

• Non-retroactivity: A grant may be awarded for a project which has already begun only where the applicant can demonstrate in the grant application the need to start the action before the grant agreement is signed. In accordance with Article 193 of the Financial Regulation, costs eligible for financing may not have been incurred prior to the date of submission of the grant application. No grant may be awarded retrospectively for a project already completed.

• Non-cumulative: In accordance with Article 191(3) of the Financial Regulation, in no circumstances shall the same costs be financed twice from the EU budget. In no circumstances shall the same costs be financed twice by the Union budget. To ensure this, the applicant shall indicate the sources and amounts of Union funding received or applied for the same project or part of the project or for its functioning during the same financial year as well as any other funding received or applied for the same project.

1.10 EFSA GRANT CONTRIBUTION

The form of grant awarded under this Call is grant based on reimbursement of a specified proportion of the total eligible project costs actually incurred (EU Financial Regulation, Article 125 (1)(f).

Lot 1
The project to be supported under this Call is co-financed by EFSA at maximum 90% of the total eligible project costs. In addition, the maximum possible amount of EFSA grant for the project is 30,000 €. In other words, the grant has double ceiling: the maximum amount and the reimbursement rate applied on the total eligible project cost.

Lot 2
The project to be supported under this Call is co-financed by EFSA at maximum 90% of the total eligible project costs. In addition, the maximum possible amount of EFSA grant for the project is 150,000 €. In other words, the grant has double ceiling: the maximum amount and the reimbursement rate applied on the total eligible project cost.

EFSA intends to fund one proposal for each lot following this Call. However, EFSA reserves the right not to award all the funds available at any cost, e.g. if the quality of submitted proposals will not be satisfactory.

Please note that EFSA reserves the right not to award any grant and/or to cancel the whole grant procedure at any time before the signature of the grant agreement without any compensation to be paid to the applicant.

The total amount of estimated eligible costs, as presented by the applicant in the estimated budget (Annex 3) (see also part 1.11), and which serves as a basis for calculation of the initial EFSA grant, will be verified by EFSA during the evaluation of proposals. EFSA reserves the right to implement the necessary adaptations to the estimated eligible costs in case the Rules on eligibility of costs (Annex 1) were not correctly applied by the applicant.

If the amount granted is lower than the funding sought by the applicant, it is up to the applicant to find supplementary financing or to reduce the total cost of the project without diluting either the objectives or the content.

1.11 ESTIMATED BUDGET AND ELIGIBLE COSTS

The proposal must be accompanied by the estimated budget (Annex 3) which must be established in line with the Rules on eligibility of costs (Annex 1). The estimated budget must show all the costs and income which the applicant considers necessary to carry out the project.

Estimated budget must be:

• sufficiently detailed to permit identification, monitoring and checking of the costs; • balanced, i.e. total income and total project costs must be equal;
• consistent with the work plan;
• expressed in Euro.

Estimated budget – cost side:6

• Eligible direct costs:

1. Costs of personnel;

2. Travel costs and subsistence allowances;

3. Depreciation costs of equipment or other assets;

4. Consumables and supplies;

5. Workshops, seminar, conferences;

6. Subcontracting;

7. Eligible VAT;

8. Miscellaneous costs are costs arising directly from the requirements imposed by the grant agreement.

The above categories represent an exhaustive list of possible eligible direct costs. However, if, for example, the project does not foresee costs for workshops / seminars / conferences, then this category of costs can be left empty in the estimated budget.

• Eligible indirect costs incurred in carrying out the project are eligible for a flat-rate funding capped at not more than 10% of the total eligible direct costs. If a beneficiary (partner in the consortium) already receives an operational grant from the EU budget its indirect costs are not eligible under the present call.

Estimated budget – income side:

• Mandatory incomes:
  1. Grant requested from EFSA;
  2. Applicant’s financial contribution; 3. Partners financial contribution;

Optional incomes:
4. Financial contributions from other public bodies;

5. Income generated by the project.

1.12 APPROVED ESTIMATED BUDGET

The estimated budget submitted with the proposal is analysed by EFSA, as part of evaluation process, in order to assess whether:

• it is realistic;

• it is consistent with the proposed project;

• the estimated budget is sufficiently detailed;

• the cost items are reasonably justified;

• to eliminate cost items which cannot be accepted according to the Rules on eligibility of costs (Annex 1).

An overestimation or underestimation of costs, or missing justification of the costs, missing details, or detected inconsistency with the technical description of the project will have a negative impact on the evaluation score under the award criterion 6.

If EFSA regards the estimated budget as realistic, consistent with the technical description of project, sufficiently detailed, well justified and established in accordance with the Rules on eligibility of costs (Annex 1) and no modification is needed, it will become the approved estimated budget and the EFSA grant may correspond to the applicant’s request. In some cases, the analysis of the estimated budget could result in EFSA suggesting reductions, e.g. need to correct the costs in line with the Rules on eligibility of costs. After the proposed modifications are agreed by the applicant and EFSA, the estimated budget, as modified, will become the approved estimated budget for the project.

1.13 INITIAL EFSA GRANT

Having agreed the approved estimated budget, and provided the proposal is selected for grant award, EFSA will establish the amount of the initial EFSA grant, having regard to the limits set out in part 1.10 of this call. The initial EFSA grant will be expressed as an amount in Euro and also as a percentage (EFSA max. 90% co-financing rate) of the total eligible project cost. This amount will be indicated in the grant agreement as the maximum grant amount.

1.14 FINAL EFSA GRANT

Maximum grant amount set out in the grant agreement was calculated based on the estimated eligible costs. The final EFSA grant will naturally have to be determined based on actually incurred costs. The final EFSA grant is determined by EFSA in line with Article II.25 of the grant agreement.

(TRUNCATED)