Analysis of Follow-up Evaluation Results of Research and Development (R&D) Projects Applying Logic Model to Elucidate the Process of Innovation Evaluation 2007（ AEA Annual Conference in Baltimore) Nov. 7, 2007 Kazuki Ogasahara 1) Osamu Nakamura 2) Kazuyuki Inahashi 1) Chikahiro Miyokawa1) Yoshitaka Kimura1) Yasukuni Okubo1)

[email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

1)Ministry of Economy, Trade and Industry, JAPAN (METI) 2)National Institute of Advanced Industrial Science and Technology (AIST)

Opinions expressed here are solely those of the authors.

1

Contents １．Background and Objectives ２．Current METI R&D Follow-up Evaluation ３．Challenge of New Follow-up Survey to Elucidate Innovation Process ４．Conclusion

2

Background of this study Evaluation Policy Strategic planning

PLAN

Ex-ante Evaluation

Implementation of R&D

Ex-post Evaluation

DO

Interim Evaluation ACTION Revised planning/ re-allocation of budget

CHECK Performance assessment

Follow-up Evaluation

Less effective for using to formulate subsequent policies and measures

Why？

3

Objectives of this study

z Look for the reasons why follow-up evaluation is less effective to apply for formulating subsequent policies and measures z Discuss new Follow-up Survey Metod

4

Contents １．Background and Objectives ２．Current METI R&D Follow-up Evaluation ３．Challenge of New Follow-up Survey to Elucidate Innovation Process ４．Conclusion

5

METI Follow-up Evaluation System

6

R&D Project Evaluations at METI Projects related to R&D

-1

1

2

3

4

5

Internal Internal ● External External External

○

○

○

○

○

6

7

8

9 10

duration (year)

Monitering Ex-ante Type of Evaluation Interim Ex-post Follow-up

●

●

●

Managing Division Technology Evaluation and Research Division

1 or 2 projects / a year ・Projects completion is five years before ・Big projects ( > around 1 billion J\) ・One project / Technical field 7

Follow-up Evaluation procedure at METI

1. Interview with responsible persons of projects ↓ 2. Survey documents by collecting projects related materials and interviews ↓ 3. Panel reviews by external evaluators ↓ 4. Evaluation report for each evaluation item with comments of evaluators

8

One Example： Follow-up Survey

9

Outcomes & Impacts Linkages Investigated at Follow-up Evaluation Study of Advanced Ceramics Project

1980Advanced ceramics ; Expected big market for new material industry 【Fine Ceramics Boom】

External Influences

（Expected to create 1 trillion \ Market ）

（Activities ）

(Project Activities） Develop fundamental & generic technologies of advanced ceramics applied for structural materials ・ Manufacturing technologies including raw material preparation, Evaluation technologies and Processing & machining technologies for Silicon carbide & its series ・Design standards for ceramics parts application

Rising negative opinions of ceramics parts application for automobile

Peak of R&D for Advanced Ceramics

(Outputs） Fundamental & generic technologies of advanced ceramics (Silicon carbide & its series） applied for structural materials

(Outcomes） Non-oxide ceramics parts Direct applica tion

・Raw material production & preparation technologies ・Molding & Sintering technologies ・Machining & Bonding technologies ・Evaluation technologies

Applied the knowhow cultivated during structural ceramics’ research & development

Applied manufacturing & evaluation technologies

(Draft translation) METI Evaluation Report, 2000 http://www.meti.go.jp/policy/tech_evaluation/ e00/03/h12/087.pdf

Advancement of metallic material technology Design policy change (low cost principle)

1990-

Applied raw material production & preparation technologies

・Ceramics raw material （powder) ・Turbo-charger rotor for automobile engine ・Ceramics parts for bearing ・Roller for steel rolling ・Ceramics parts for semiconductor manufacturing ・Ceramic parts for pump ・Ceramics parts for machining facilities & Evaluation equipment

In the future

・Promoting energy saving ・Discover new merit of ceramics with high temperature resistance & high rigidity

（Possibility ）

If the ceramics fracture toughness property can improve as well as metal and the manufacturing technologies to produce ceramics parts more low cost ｃａｎ be realized , structural ceramics may apply for more various fields of industry （Difficult to estimate the time realized)

Ceramics application projects

・Ceramics gas turbine project ・Synergy ceramics project

Functional Ceramics

・Ceramics sensors ・Substrate for power module circuit

Ceramics parts

・ZrSi2 sputtering target for glass surface treatment ・ Particulate Filters for diesel engine & Ceramic catalyst carrier for automobile engine （cordierite） ・Ceramics filter

Ceramics particle & parts

・Ink-jet printer ink (dispersion medium) ・Piezoelectric material ・Solid electrolyte fuel cell etc.

Outcomes of R&D outputs （Red letter； Practically applied) 10

Outcomes & Impacts Linkages Investigated at Follow-up Evaluation Study of Advanced Ceramics Project

1980External Influences

Advanced ceramics ; Expected big market for new material industry 【Fine Ceramics Boom】

（Expected to create 1 trillion \ Market ）

1990Peak of R&D for Advanced Ceramics

Advancement of metallic material technology Design policy change (low cost principle) Rising negative opinions of ceramics parts application for automobile

（Activities (Outputs） (Outcomes） ） Advanced ceramics ; Expected big (Project Activities） Develop fundamental & generic technologies of advanced ceramics applied for structural materials

Fundamental & generic technologies of advanced ceramics (Silicon carbide & its series） applied for structural materials

Non-oxide ceramics parts

・Ceramics raw material （powder) ・Turbo-charger rotor for automobile engine ・Ceramics parts for bearing ・Roller for steel rolling ・Ceramics parts for semiconductor manufacturing ・Ceramic parts for pump ・Ceramics parts for machining facilities & Evaluation equipment

In the future

・Promoting energy saving ・Discover new merit of ceramics with high temperature resistance & high rigidity

（Possibility ）

If the ceramics fracture toughness property can improve as well as metal and the manufacturing technologies to produce ceramics parts more low cost ｃａｎ be realized , structural ceramics may apply for more various fields of industry （Difficult to estimate the time realized)

market for new material industry Fundamental & generic Develop fundamental & generic 【Fine Ceramics Boom】 technologies of advanced technologies of advanced （Expected to create ceramics (Silicon ceramics applied for structural 1 trillion \ Market ） carbide & its series） applied for structural materials ・ Manufacturing technologiesmaterials ・Raw material production & including raw material preparation, Outcomes of R&D Evaluation technologies and preparation technologies outputs ・Molding & Sintering technologies Processing & machining （Red & Bonding technologies for Silicon carbide・Machining & letter； technologies its series Practically ・Evaluation technologies ・Design standards for ceramics applied) parts application ・ Manufacturing technologies including raw material preparation, Evaluation technologies and Processing & machining technologies for Silicon carbide & its series ・Design standards for ceramics parts application

・Raw material production & preparation technologies ・Molding & Sintering technologies ・Machining & Bonding technologies ・Evaluation technologies

Applied the knowhow cultivated during structural ceramics’ research & development

Applied manufacturing & evaluation technologies

Applied raw material production & preparation technologies

Ceramics application projects

・Ceramics gas turbine project ・Synergy ceramics project

Functional Ceramics

・Ceramics sensors ・Substrate for power module circuit

Ceramics parts

・ZrSi2 sputtering target for glass surface treatment ・ Particulate Filters for diesel engine & Ceramic catalyst carrier for automobile engine （cordierite） ・Ceramics filter

Ceramics particle & parts

・Ink-jet printer ink (dispersion medium) ・Piezoelectric material ・Solid electrolyte fuel cell etc.

11

Outcomes & Impacts Linkages Investigated at Follow-up Evaluation Study of Advanced Ceramics Project

1980Advanced ceramics ; Expected big market for new material industry 【Fine Ceramics Boom】

（Activities ）

（Expected to create 1 trillion \ Market ）

(Project Activities） Develop fundamental & generic technologies of advanced ceramics applied for structural materials ・ Manufacturing technologies including raw material preparation, Evaluation technologies and Processing & machining technologies for Silicon carbide & its series ・Design standards for ceramics parts application

1990Peak of R&D for Advanced Ceramics

(Outputs）

Non-oxide ceramics parts ・Ceramics raw material （powder) ・Turbo-charger rotor for automobile engine ・Ceramics parts for bearing ・Roller for steel rolling ・Ceramics parts for semiconductor manufacturing ・Ceramic parts for pump ・Ceramics parts for machining facilities & Evaluation equipment

Rising negative opinions of ceramics parts application for automobile

(Outcomes）

Fundamental & generic technologies of advanced ceramics (Silicon carbide & its series） applied for structural materials ・Raw material production & preparation technologies ・Molding & Sintering technologies ・Machining & Bonding technologies ・Evaluation technologies

Advancement of metallic material technology Design policy change (low cost principle)

Non-oxide ceramics parts ・Ceramics raw material

Direct （powder) Direct applic ・Turbo-charger rotor for ation automobile engine application ・Ceramics parts for bearing

Applied the knowhow cultivated during structural ceramics’ research & development

Applied manufacturing & evaluation technologies

Applied raw material production & preparation technologies

・Roller for steel rolling ・Ceramics parts for semiconductor manufacturing ・Ceramic parts for pump ・Ceramics parts for machining facilities & Evaluation equipment

In the future

・Promoting energy saving ・Discover new merit of ceramics with high temperature resistance & high rigidity

（Possibility ）

If the ceramics fracture toughness property can improve as well as metal and the manufacturing technologies to produce ceramics parts more low cost ｃａｎ be realized , structural ceramics may apply for more various fields of industry （Difficult to estimate the time realized)

Ceramics application projects

・Ceramics gas turbine project ・Synergy ceramics project

Functional Ceramics

・Ceramics sensors ・Substrate for power module circuit

Outcomes Ceramics application of R&D Ceramics parts outputs projects ・Ceramics gas （Red turbine letter； project Ceramics particle & parts Practically ・Synergy ceramics project applied)

・ZrSi2 sputtering target for glass surface treatment ・ Particulate Filters for diesel engine & Ceramic catalyst carrier for automobile engine （cordierite） ・Ceramics filter

・Ink-jet printer ink (dispersion medium) ・Piezoelectric material ・Solid electrolyte fuel cell etc.

12

Outcomes & Impacts Linkages Investigated at Follow-up Evaluation Study of Advanced Ceramics Project

1980External Influences

External Influences

Advanced ceramics ; Expected big market for new material industry 【Fine Ceramics Boom】

（Activities ）

（Expected to create 1 trillion \ Market ）

(Project Activities） Develop fundamental & generic technologies of advanced ceramics applied for structural materials ・ Manufacturing technologies including raw material preparation, Evaluation technologies and Processing & machining technologies for Silicon carbide & its series ・Design standards for ceramics parts application

Advancement of metallic material technology Design policy change (low cost principle)

1990-

Rising negative opinions of ceramics parts application for automobile

Peak of R&D for Advanced Ceramics

(Outputs） Fundamental & generic technologies of advanced ceramics (Silicon carbide & its series） applied for structural materials

(Outcomes） Non-oxide ceramics parts Direct applic ation

・Raw material production & preparation technologies ・Molding & Sintering technologies ・Machining & Bonding technologies ・Evaluation technologies

Applied the knowhow cultivated Applied the knowhow cultivated during structural during structural ceramics’ research ceramics’ research & development & development Applied Applied & manufacturingmanufacturing & evaluation technologies evaluation technologies

Applied raw materialApplied raw material production & production & preparation technologies preparation technologies

・Ceramics raw material （powder) ・Turbo-charger rotor for automobile engine ・Ceramics parts for bearing ・Roller for steel rolling ・Ceramics parts for semiconductor manufacturing ・Ceramic parts for pump ・Ceramics parts for machining facilities & Evaluation equipment

Ceramics application projects

In the future

・Promoting energy saving ・Discover new merit of ceramics with high temperature resistance & high rigidity

（Possibility ）

If the ceramics fracture toughness property can improve as well as metal and the manufacturing technologies to produce ceramics parts more low cost ｃａｎ be realized , structural ceramics may apply for more various fields of industry （Difficult to estimate the time realized)

Functional Ceramics If the ceramics fracture ・Ceramics sensors Ceramics parts toughness property can improve ・ZrSi2 sputtering ・Substrate for power as well as metal and target the for glass module circuit surfaceOutcomes treatment manufacturing technologies to & Ceramics particle of R&D ・ Particulate Filters produce ceramicsparts parts by more for diesel engineink & outputs ・Ink-jet printer low cost can be (dispersion realized , medium) Ceramic catalyst （Red carrier for automobile structural ceramics mayletter； apply for ・Piezoelectric engine（cordierite） Practically material more various fields of industry ・Ceramics filter ・Solid electrolyte applied) （Difficult to estimate the time fuel cell etc. 13 realized) ・Ceramics gas turbine project ・Synergy ceramics project

Functional Ceramics

・Ceramics sensors ・Substrate for power module circuit

Ceramics parts

・ZrSi2 sputtering target for glass surface treatment ・ Particulate Filters for diesel engine & Ceramic catalyst carrier for automobile engine （cordierite） ・Ceramics filter

Ceramics particle & parts

・Ink-jet printer ink (dispersion medium) ・Piezoelectric material ・Solid electrolyte fuel cell etc.

Features of logic model for follow-up evaluation in METI

Only realized linkages outputs and effects are drawn. No respective among outputs and Effective information for confirming Outputs are usually linked directoryproject to outcomes. Technical elements which connect 1980In the future 1990commercialized products effects and impacts ; Accountability to the External linkages are not explicit Influences public （Activities (Outputs） (Outcomes） （Possibility Advancement of metallic material technology Design policy change (low cost principle)

Advanced ceramics ; Expected big market for new material industry 【Fine Ceramics Boom】

（Expected to create 1 trillion \ Market ）

）

(Project Activities） Develop fundamental & generic technologies of advanced ceramics applied for structural materials ・ Manufacturing technologies including raw material preparation, Evaluation technologies and Processing & machining technologies for Silicon carbide & its series ・Design standards for ceramics parts application

Rising negative opinions of ceramics parts application for automobile

Peak of R&D for Advanced Ceramics

Fundamental & generic technologies of advanced ceramics (Silicon carbide & its series） applied for structural materials

Non-oxide ceramics parts

Direct applica tion

・Raw material production & preparation technologies ・Molding & Sintering technologies ・Machining & Bonding technologies ・Evaluation technologies

Applied the knowhow cultivated during structural ceramics’ research & development

z Missing relevant linkage (Draft translation) z METI Missing middle Evaluation Report, 2000 http://www.meti.go.jp/policy/tech_evaluation/ (outcomes) e00/03/h12/087.pdf

Applied manufacturing & evaluation technologies

Applied raw material production & preparation technologies

・Ceramics raw material （powder) ・Turbo-charger rotor for automobile engine ・Ceramics parts for bearing ・Roller for steel rolling ・Ceramics parts for semiconductor manufacturing ・Ceramic parts for pump ・Ceramics parts for machining facilities & Evaluation equipment

・Promoting energy saving ・Discover new merit of ceramics with high temperature resistance & high rigidity

）

If the ceramics fracture toughness property can improve as well as metal and the manufacturing technologies to produce ceramics parts more low cost ｃａｎ be realized , structural ceramics may apply for more various fields of industry （Difficult to estimate the time realized)

Ceramics application projects

・Ceramics gas turbine project ・Synergy ceramics project

Functional Ceramics

・Ceramics sensors ・Substrate for power module circuit

Ceramics parts

・ZrSi2 sputtering target for glass surface treatment ・ Particulate Filters for diesel engine & Ceramic catalyst carrier for automobile engine （cordierite） ・Ceramics filter

Ceramics particle & parts

・Ink-jet printer ink (dispersion medium) ・Piezoelectric material ・Solid electrolyte fuel cell etc.

Outcomes of R&D outputs （Red letter； Practically applied) 14

Features of logic model for follow-up evaluation in METI 【Evaluators’ comments for the In the future 19801990unrealized impacts of 1 trillion \ Market】 １．Opinion change: Worth the cost, not performance Outcomes of ２．Characteristics ofofceramics is No description the expected outcomes R&D outputs （Red letter； Practically itself ductile and still have not applied) which are necessary to make up impact realized the material which satisfy If the ceramics fracture the customers’ request of 1 trillion toughness property can improve \ market as well as metal and the manufacturing technologies to ３． Technologies of cost reduction have not reached aimed target toproduce ceramics parts more low cost ｃａｎ be realized , Missing outcomes realized 1 trillion \ market structural ceramics may apply for ４．Contending technology expected more various field of industry advancing such as metal and （Difficult to estimate the time ceramic material of oxide systemrealized) External Influences

Advancement of metallic material technology Design policy change (low cost principle)

Advanced ceramics ; Expected big market for new material industry 【Fine Ceramics Boom】

（Expected to create 1 trillion \ Market ）

(Outpus）

Develop fundamental & (Project Activities） generic technologies of advanced ceramics applied for structural materials ・ Manufacturing technologies including raw material preparation, Evaluation technologies and Processing & machining technologies for Silicon carbide & its series ・Design standards for ceramics parts application

Rising negative opinions of ceramics parts application for automobile

Peak of R&D for Advanced Ceramics

Fundamental & generic technologies of advanced ceramics (Silicon carbide & its series） applied for structural materials

(Applied technologies・commercialized products） Non-oxide ceramics parts

・Raw material production & preparation technologies ・Molding & Sintering technologies ・Machining & Bonding technologies ・Evaluation technologies

・Ceramics raw material （powder) ・Turbo-charger rotor for automobile engine ・Ceramics parts for bearing ・Roller for steel rolling ・Ceramics parts for semiconductor manufacturing ・Ceramic parts for pump ・Ceramics parts for machining facilities & Evaluation equipment

・Promoting energy saving ・Discover new merit of ceramics with high temperature resistance & high rigidity

（Possibility）

If the ceramics fracture toughness property can improve as well as metal and the manufacturing technologies to produce ceramics parts more low cost ｃａｎ be realized , structural ceramics may apply for more various field of industry （Difficult to estimate the time realized)

Ceramics application projects

Applied the knowhow cultivated during structural ceramics’ research & development

Applied manufacturing & evaluation technologies

Applied raw material production & preparation technologies

・Ceramics gas turbine project ・Synergy ceramics project

Functional Ceramics

・Ceramics sensors ・Substrate for power module circuit

Ceramics parts

・ZrSi2 sputtering target for glass surface treatment ・ Particulate Filters for diesel engine & Ceramic catalyst carrier for automobile engine （cordierite） ・Ceramics filter

Ceramics particle & parts

・Ink-jet printer ink (dispersion medium) ・Piezoelectric material ・Solid electrolyte fuel cell etc.

15

Summary of past survey results in METI

Effective information for confirming project effects and impacts ; Accountability to the public But

z Missing relevant linkage z Missing middle z Missing outcomes expected

→ How to solve? 16

Contents １．Background and Objective ２．Current METI R&D Follow-up Evaluation ３．Challenge of New Follow-up Survey to Elucidate Innovation Process ４．Conclusion

17

Challenge of New Follow-up Survey to Elucidate Innovation Process

How to solve? I will challenge how to solve in the view point of one engineer who tried to developed engine valves for automobile but failed.

18

Case Study Trial for Advanced Ceramics Project Expected impact (１trillion J\ market) Turbo-charger rotor for automobile engine

Develop base technologies of advanced ceramics for structural material

Base technologies

Raw material preparation

Turbo-charger rotor for automobile engine

Ceramic valve for automobile engine

Molding Sintering Machining & Grinding Bonding

Ceramic valve for automobile engine

Evaluation/Inspection

1. Describe the outcomes which will make up expected impact (１trillion J\ market) 2. Select one pair of outcome which has realized and unrealized 3. Case Study：outcome realized vs. outcome expected

19

Case Study Trial for Advanced Ceramics Project ； Simple model for the reference for explanation of our trial Expected impact (１trillion J\ market) Turbo-charger rotor for automobile engine Raw material preparation Molding (Injection)

Develop base technologies of advanced ceramics for structural material

Sintering

Raw material preparation

Different Ceramic valveSpecifications for automobile engine

Molding

Raw material preparation

Sintering

Molding ( Injection casting or press molding)

Machining & Grinding

Sintering

Bonding

Machining & Grinding

Machining & Grinding Evaluation/Inspection

Automobile ・Production ・Price ・Quality Given the specification of each technology

Evaluation/Inspection Advance base technology to create the following outcome ¾ In the case study analysis, the technical factors which are necessary to create each outcome should be considered. ↓ Evaluation/Inspection

¾ Parts are applied for automobile. So, specification of each technical factor is Applicable? decided by the request of automobile. ¾ The progress in one outcome to the following outcome, which are connected by arrow, should be evaluated in the view of these technologies factors as follows;

20

Case Study Trial for Advanced Ceramics Project ； Simple model for the reference for explanation of our trial Expected impact (１trillion J\ market) Turbo-charger rotor for automobile engine Raw material preparation

Automobile parts:

Molding (Injection)

Develop base technologies of advanced ceramics for structural material

Sintering Machining & Grinding Evaluation/Inspection

Raw material preparation

Ceramic valve for automobile engine

Molding Sintering

Raw material preparation

Applicable?

Molding ( Injection casting or press molding)

Machining & Grinding

Sintering

Bonding

Machining & Grinding

Evaluation/Inspection

Specifications requested for the elemental technologies to realize outcome ・Production ・Price ・Quality

Evaluation/Inspection

（Comparison； authors’ opinion） Basic Technologies developed

Applicability of basic technology developed Turbo-charger rotor

Ceramic valve （my opinion comparing with rotor)

Raw material preparation

○

×

Molding

○

×

Sintering

○

△

・Higher fracture toughness property ・Cost-reduction： Near-net-shape molding, ・ Machining before sintering, ・Automatic machining & grinding system, ・Automatic inspection system

Machining & Grinding

○

△

Need mass production system

Evaluation

○

△

Higher reliability

21

Case Study Trial for Advanced Ceramics Project ； Simple model Expected impact (１trillion J\ market) Turbo-charger rotor for automobile engine Raw material preparation

Automobile:

Molding (Injection)

Develop base technologies of advanced ceramics for structural material

Sintering Machining & Grinding Evaluation/Inspection

Raw material preparation

Near-netshape molding for automobile valve

Molding Sintering

Ceramic valve for automobile engine Raw material preparation Molding ( Injection casting or press molding)

Machining & Grinding

Molding

Bonding

Raw material preparation

Machining & Grinding

Machining before sintering

Evaluation/Inspection

Evaluation/Inspection

Sintering

（Case study example； authors’ opinion）Basic Technologies

Specifications requested for the elemental technologies to realize outcome ・Production ・Price ・Quality

Applicability of basic technology developed

This R&D developed

activity is needed forCeramic valve （my opinion comparing with rotor) Turbo-charger rotor advancing technologies ・Higher fracture toughness property ○

△

○

×

Sintering

○

△

・Cost-reduction： Near-net-shape molding, ・ Machining before sintering, ・Automatic machining & grinding system, ・Automatic inspection system

Machining & Grinding

○

△

Need mass production system

Evaluation

○

△

Higher reliability

Raw material preparation Molding

More survey of this ‘Missing Middle’

22

Challenging follow-up survey

１．Develop logic model with relevant linkages to the outcomes which will make up expected impacts ２．Select outcome linkages for ‘Case study’ : outcome realized vs. outcome expected ３．‘Case study’ based on element technologies 23

Case study protocols Item

Conventional

Additional

1Reasons for involving this project

・What the purpose of Bonding this project?

・ What’s your business target and what’s the (element ) technologies did you create ? ・ What’s outcomes do you plan to make up impacts? ・What’s kind of influences of this project on R&D results of your company in the vie point of element technology?

2. Project activities

・ What’s kind of influences of this project on R&D results of your company?

3. R&D results

・What’s kind of influences of this project influence on the following items of your company ？; - R&D portfolio - R&D strategy

・ What’s kind of influences of this project on the following items for your company in the view point of element technology？; -R&D portfolio

4. R&D impacts

・ What’s kind of influences of this project on your R&D? Let me see the successful ( or unsuccessful ) results (technologies developed, products or productive process) , and their history to them?

・ What’s your comments on this logic model? - another outcomes of missing middle - another outcomes to make up impact ・What’s the unsuccessful reasons in the viewpoint of element technologies of outcome？ ・Case study questionnaires

5.Business impacts

What’s kind of influences of this project on the success of your business?

・What’s kind of influences of this project on the success of your business in the view point of element technologies of outcome?

6.Lessons of this project

What’s is your lesson (i.e. on management) of this projects 24

Contents １．Background and Objective ２．Current METI R&D Follow-up Evaluation ３．Challenge of New Follow-up Survey to Elucidate Innovation Process ４．Conclusion

25

Conclusion z Follow-up survey is essential for PDCA cycle z Effective follow-up survey method require; ・ Comprehensive logic model with relevant linkage ・ Case study： - Outcome realized vs. outcome expected - Based on element technologies which make up outcome 26

Next step z We’ll try to apply this method for the follow-up evaluation survey z Launched the development of the supporting system which analyzes the information of Web to investigate outcomes relationship and element technologies which make up the outcome as follows; 27

General system flow of supporting system Measurement of the impact of technology by structuring knowledge from the Web. Inputs

・PJ organization, researcher name ・R&D outputs (element technologies) ・Direct outcomes (planed/actual) ・Direct customer （ organization ） ・Expected outcomes, impacts （ organization ）

１ｓｔ Step

Web

In the future

Patents DB Tech. DB

Analyzed Result ・Outcome linkage with element technologies ・Network of organization (or researcher) etc.

Private communication with Dr. Yutaka Matsuo, Institute of Engineering Innovation, The University of Tokyo Center for Knowledge Structuring,The University of Tokyo

28

Social Network Mining • There is vast amount of information on the Web. • We can identify relations among entities; e.g., – social relations among researchers; co-authorship, coattendance, co-affiliation – social relations among corporations; alliance, lawsuit – relation between concepts; is-a, part-whole – and many other relations; causal relation, functional relation…

• We are developing Web mining methods to extract relations among entities, as well as precisely identifying entities. – Combined with IR, natural language processing, and machine learning. – Mainly in WWW and Semantic Web field.

Private communication with Dr. Yutaka Matsuo, Institute of Engineering Innovation, The University of Tokyo Center for Knowledge Structuring,The University of Tokyo

29

Logic model (structurizing the technologies linkages） Applied research

Project

Direct outcome

Impact

Element tech. A

Input

Industrial application (placing on the market）

Element tech. B Name

Outputs

Organization

Element tech. C

Element tech. C

Element tech. F

Element tech. D Tech.１

Element tech. A

Element tech. G

Element tech. E Element tech. B Element tech. C Element tech. D

Activities

Tech.2

Element tech. E

Element. H Element tech. F Name・organization Name・Organization

Outcome

Expansion R&D network or Applied research in this field

Element tech. H Element tech. I Element tech. J Element tech. K

Element tech. C Element tech. D Element tech. E Element tech. K Name・Organization

30

Simplified logical technologies(outcomes) linkages to realize hydrogen fuel cell vehicle Solid polymer battery （for automobile ） Specified more to produce the following outcome

hydrogen fuel cell vehicle

Stack Electrolyte film

Electric motor

Electrode catalyst

Solid polymer battery

battery separator

General basic technologies

Hydrogen storage tank

High performance Inverter

Electrolyte film Electrode catalyst

Supply of hydrogen

?

Hydrogen storage tank （for automobile ）

Safety technology for hydrogen

Hydrogen storage body

Metal bonding technologies （prevent from hydrogen embrittlement

Hydrogen storage alloy Structure design

Design technologies for hydrogen storage tank Guidance for tank design Evaluation method & standard

Pressure vessel Absorption・ desorption system

∬

Hydrogen storing alloy

In this logic model ‘technology, systems or products)’ consists of technical elements and is the consolidated unit size which can be evaluated the target outcomes contribution The specification requested for elemental technologies controlled by the requested conditions (i.e. needs etc.) by the following ‘technology’. (Technology , system or product) (Elemental technology)

31