PREVIEW ONLY - No response can be submitted.
Back to Library
Customize
Verification and Validation in Scientific Simulations
Project Overview
General information
What is your position in a project (general, connected with the university position)?
*
Research/Project Assistant
Project Leader/Supervisor (mostly managing, controlling than executing)
Other
Estimate your experience in large scale simulation projects (in months, 1 month:= 6h*4(days)*4(weeks) = 96h)
*
< 6
6 - 24
> 24
Large scale simulation can be considered as computational simulation consisted on tens, hundreds, thousands or more equations to solve.
Do you programm?
*
Yes
No
For how long have you programmed? (in months, 1 month:=96h)
*
< 6
6 - 24
> 24
Which computer languages do you use?
*
Fortran
C/C++
Java,C#
Script languages (e.g. Python, TCL, Ruby)
Mathematical programming languages (e.g. Mathlab, Octave, Mathematica, Maple)
Other
How many hours of courses (programming courses, software engineering related topics) have you taken up?
*
0
15 - 30
31 - 90
> 90
Projects overview
What are the reasons for your project?
*
Part of my work
For science needs
For commercial needs
Other
What are general goals of your project?
*
Invention
Improvement
Evaluation
Other
Choose general types of projecs that you have been involved in:
*
Modelling/ Designing projects - project outcome is a design of a construction/ real-world element
Software projects - project outcome is an application
Projects based on experiments - project deliverable is an experiment environment and its results
Analytic projects - project deliverable is an analytical solution (including methods and results)/ mathematical model
None of them, other type
How many people at the university are involved in your project?
*
1 person - just me
2 people: me and sb else (supervisor or project assistant)
3 people and more
Requirements analysis
Planning
Building a model/ prototype/ solution/ construction
Experiments
Programming
Verifying, validating, testing
Other activities (e.g. documenting, meetings)
Project Timetable
Do you plan your (or your team's) project work?
*
Yes, always
Yes, sometimes
Yes. Although I get the plan, I also plann my work by myself
No, I get the plan prepared
Very rarely
No
Why don't you prepare your project plan?
Do you write your plan down?
*
Yes
No
Is there a project plan review (with supervisor/ project leader) to make sure that it is adequate and completed?
*
Yes
No
Do you have a requirements specifications for your conceptual model and for your numerical/simulation model?
*
Yes, very detailed
Yes, but it is general
Sometimes
Very rarely
No
What issues does your requirements specification include?
*
Description of the intended behaviour of the numerical algorithm when it runs
Specification of all conditions under which the model should work correctly
Identification of the simualtion model functions and interactions
Specification of input and output for the numerical model/algorithm
Software configuration (e.g. list of tools, technologies, computer languages)
Hardware configuration (e.g. properties of hardware)
None of them, other
Which of the following descriptions of requirements specification is the most suitable for your model requirements specification?
*
It contains mostly plain text
It is text with many diagrams, graphs or graphics
Why don't you have a requirements specification for your models?
Do you use any version controll system (e.g. CVS, SVN) during your work?
*
Yes
No
Verification and Validation Processes
Verification and Validation Plan
Does your project plan include V&V plan?
*
Yes, and it is a detailed specification
Yes, but it is a general plan
Sometimes
Very rarely
No, because I don't verify or validate my models/software/results
No, because I usually verify or validate with the same methods
No, for other reasons
What issues does your V&V plan include?
*
List of key points to verify
List of key points to validate
List of experiments to carry out
Description of errors to estimate
Description of V&V methods
Benchmarks and samples to test code
Software or hardware configuration (e.g. properties of hardware, list of tools/ technologies) to help V&V processes
Other
Why don't you have a V&V specification for your project?
Verification and Validation Activities
Solution verification
Do you estimate the numerical accuracy of the given solution?
*
Yes
No
How do you estimate the accuracy of the solution?
*
Comparison of numerical solutions of the discretized equations on different grid sizes (related to h-adaptivity)
Comparison of numerical solutions from different discretization methods on the same grid (related to p-adaptivity)
Other
Why don't you estimate the accuracy of the solution?
Which of the following errors do you estimate?:
*
Discretization error (related to the numerical method used to solve the problem)
Iterative error (related to approximation error)
Geometry approximations errors
Other
None
Which of the following steps do you perform for nondeterministic simulations?
*
Probability distribution for unknown or experimentally uncontrolled values
Selecting input values using statistical sampling procedures (e.g. Monte Carlo)
Probability distribution of the required output quantities from multiply individual computations
Nothing, just waiting for experiment results
Other
Not applicable
Code Verification
Numerical Model/Algorithm Verification
How do you verify implemented numerical model/ algorithm?
*
Analytic solutions for simplified physics
Method of manufactured solution
Benchmark solutions
Other
Describe your other method:
Software Quality Analysing and Testing
Which of the following static analysis activities do you do?
*
Syntax Analysis (loops; conditions; variables types; function inputs-outputs types)
Data Analysis (data dependiency; data flow by making a flow diagram and analysing it)
Cause-Effect graphing
Controll analysis (calling structure; concurrent process; control flow; state transition)
Interface analysis (model interface; user inteface)
Fault/Failure analysis
None of them
Which of the following dynamic analysis activities do you do?
Unit tests (e.g. assertion checking)
Black Box tests (functional tests): output examining under a variety of input parameters
White Box tests (structural tests): analysing of conditions, data flow, loops, paths
Special Input testing (examining the boundary values, extreme/ invalid/ real-time/ self-driven input)
Debugging
Printing the input parameters at the end of the simulation, to be sure that they are not changed
Bottom-Up testing
Fault/Failure Insertion testing
Visualization/Animation of running program
Graphical Comparisons
Sensitivity analysis
Interface testing (data; model; user)
Object-Flow testing
Printing out a wide variety of output statistics
Complaince testing (authorisation; security; standards; performance)
None of them
Software Quality Assurance Practises
Is your code as self-documenting as possible? (precise definition of every variable used and a general description of the purpose of each major section of code)?
*
Yes
No
Is your code checked by someone else than you?
*
Yes
No
Do you use Design Patterns?
*
Yes
No
Do you use iterative methodologies for programming?
*
Yes
No
Iterative and incremental methodology is a software development process. Whole process is divided in several steps, which are standalone and delivere a part of functionality of your software. Examples of methodologies: XP, SCRUM, Agile and many more adopted to project needs.
Do you write a test first before writing the code?
*
Yes
No
Do you learn other programming technologies or languages at least once a year?
*
Yes
No
Conceptual Model Validation
Which of the following activities do you do?
*
Building the conceptual model basing on empirical data
Theoretical derivation
Personal Communication (asking people whether the model and/or its behaviour is reasonable; it requires flow chart, graphical model)
Applying accepted approaches, making rational decisions while building and judging the model
Validation with some historical data
Applying "positive economics" (:= an approach that requires only the model to be able to predict the future effects and is not concerned with model assumptions and structure)
Traces (following specific entities of model to determine if the logic is correct and if the necessary accuracy is obtained)
Other
None of them
Numerical Model Validation
What kind of experiments do you carry out?
*
Experiments that are conducted primarily to improve the fundamental understanding of some phisical process. Sometimes these are referred to as physical-discovery experiments
Experiments that are conducted primarilyfor construction or improving mathematical models of fairly well-understood phisical processes
Experiments that determine or improve reliability, performance, or safety of components, subsystems, or complete systems
Experiments that are designed, executed, and analysed for the purpose of quantitatively determining the ability of a mathematical model and its embodiment (=representative model) in a computer code to simulate a well- characterized physical process
I don't do experiments
Do you compare the numerical model input-output transformation to corresponding input-output transformations for the real system by using benchmarks?
*
Yes
No
Why don't you use benchmarks for validation?
What is the most difficult in your project to verify or validate?
*
To validate models which integrate many different components, so a qualitative dependency on input parameters is not obvious
To verify modelling results which are the solution of coupled processes which you can't decouple. There is no sure, where the deviation come from.
To verify whether the assumptions and simplifications which were done while modelling, are justified or not
To verify accuracy of the computer model
To validate The numerical formulation underlying physical model
Other
I have no problems with V&V
Describe other difficulity:
Do you document your V&V results (including testing results if you programm) ?
*
Yes
No
Do you write a V&V summary report for your supervisor/customer/project leader (or do you get such a report from your team if you are a supervisor)?
*
Yes
No
Data for Verification and Validation
Data for V&V
How do you acquire data for V&V use?
*
From the Internet
From universities' repositories
From libraries
From specialistic benchmark databases
From a customer/company
I prepare data by myself using free/commercial solvers
Other
If possible, please, give the address of the benchmark database:
Finding data for comparison tests for your engineering's problems is in your opinion:
*
Very difficult
Rather difficult
Rather easy
Very easy
Have you ever had to rewrite numerical data (e.g. from tables) from articles and other sources to prepare data for V&V?
*
Yes
No
Have you ever had to reproduct graphs, figures, grids etc. from publications and other sources in order to compare them with your results and publish those comparisons?
Yes
No
If the data (e.g. graphs, tables) was available electronically (on-line) in editable formats how much it would help you with work?
*
Unearthly
Very
Without special influence
How often, in your opinion, scientific publications don't contain complete information for equivalent reproduction of the presented results?
*
Very often
Often
Sometimes
Hardly ever
Never
Have you ever had a situation, that your simulation results were completely different from those presented in the literature?
*
Yes
No
Have you ever had to write a program/script for format conversion: from data format published in a paper to a format which is used by your tools/software?
*
Yes
No
How difficult is writing programs for format conversion?
Very difficult
Rather difficult
Rather easy
Very easy
Do you prepare verification/validation benchmarks by yourself?
*
Yes
No
Where do you keep your benchmarks?
*
On my computer only
I public them on my/university website only
I put them in some on-line official repositories/databases
What should a benchmark solution contain to be the most useful to you?
*
Conceptual description
Mathematical description
Accuracy assessment
Experimental description
Description of uncertain quantification of benchmark measurements
Code/scripts used for solved problems
Figures, tables etc. in editable formats
Other
Describe other item that a benchmark should contain:
Please, once again estimate your time (in %) in last finished project spent on V&V activities (please don't try to recreate your previous answer) :
*
Free text area for your opinion, suggestions and other statements, if you wish.
Back to Library 
Customize 