Resulaten 2 temperaturen
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
|
||||
|
||||
|
||||
python3 main.py standalone 10 20 Few-shot-learning 0.2
|
||||
python3 main.py standalone 15 16 Few-shot-learning 1.3
|
||||
|
||||
#
|
||||
python3 main.py results Few-shot-learning_1.3temp
|
||||
#python3 main.py results Few-shot-learning_1.3temp
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:47:33
|
||||
* UniqueID: 1534c754-74da-4a28-9802-e367fb8d2625
|
||||
* Gherkinscenario name: Gherkinscenario 10: Logging out the user doesn't log out my admin account_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA impersonates the customer "remnant@london.uk"
|
||||
When: UserA logs out from SystemA
|
||||
Then: UserA should still be able to access the administration dashboard
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA impersonates the customer "remnant@london.uk"
|
||||
When: UserA logs out from SystemA
|
||||
Then: UserA should still be able to access the administration dashboard
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should While UserA logs out from SystemA SystemA be able to access the administration dashboard
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'out' (line : 3 column : 31)
|
||||
ERROR:missing ',' at 'SystemA' (line : 3 column : 40)
|
||||
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 48)
|
||||
ERROR:missing '.' at 'be' (line : 3 column : 56)
|
||||
ERROR:no viable alternative at input 'able' (line : 3 column : 59)
|
||||
ERROR:missing '.' at 'access' (line : 3 column : 67)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 74)
|
||||
ERROR:missing '.' at 'administration' (line : 3 column : 78)
|
||||
ERROR:no viable alternative at input 'dashboard' (line : 3 column : 93)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'logs'. (line : 3 column : 26)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 20)
|
||||
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 56)
|
||||
ERROR:Couldn't resolve reference to Actor 'access'. (line : 3 column : 67)
|
||||
ERROR:Couldn't resolve reference to Actor 'administration'. (line : 3 column : 78)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 78
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,1
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 80
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 69.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:46:30
|
||||
* UniqueID: fd36f0e7-c22a-413d-9557-516a588f97c0
|
||||
* Gherkinscenario name: Gherkinscenario 10: Logging out the user doesn't log out my admin account
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA impersonates the customer "remnant@london.uk"
|
||||
When: UserA logs out from SystemA
|
||||
Then: UserA should still be able to access the administration dashboard
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA impersonates the customer "remnant@london.uk"
|
||||
When: UserA logs out from SystemA
|
||||
Then: UserA should still be able to access the administration dashboard
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
Given UserA should already impersonate the customer "remnant@london.uk" While UserA logs out from SystemA, UserA must still be able to access the administration dashboard.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -5
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'should' (line : 3 column : 13)
|
||||
ERROR:no viable alternative at input 'already' (line : 3 column : 20)
|
||||
ERROR:missing '.' at 'impersonate' (line : 3 column : 28)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 40)
|
||||
ERROR:missing '.' at 'customer' (line : 3 column : 44)
|
||||
ERROR:no viable alternative at input '"remnant@london.uk"' (line : 3 column : 53)
|
||||
ERROR:no viable alternative at input 'While' (line : 3 column : 73)
|
||||
ERROR:no viable alternative at input 'out' (line : 3 column : 90)
|
||||
ERROR:extraneous input 'SystemA' expecting ',' (line : 3 column : 99)
|
||||
ERROR:no viable alternative at input 'still' (line : 3 column : 119)
|
||||
ERROR:no viable alternative at input 'be' (line : 3 column : 125)
|
||||
ERROR:missing '.' at 'able' (line : 3 column : 128)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 133)
|
||||
ERROR:no viable alternative at input 'access' (line : 3 column : 136)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 143)
|
||||
ERROR:missing '.' at 'administration' (line : 3 column : 147)
|
||||
ERROR:no viable alternative at input 'dashboard' (line : 3 column : 162)
|
||||
ERROR:Couldn't resolve reference to Actor 'Given'. (line : 3 column : 1)
|
||||
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 13)
|
||||
ERROR:Couldn't resolve reference to Actor 'impersonate'. (line : 3 column : 28)
|
||||
ERROR:Couldn't resolve reference to Actor 'customer'. (line : 3 column : 44)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'logs'. (line : 3 column : 85)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 79)
|
||||
ERROR:Couldn't resolve reference to Actor 'still'. (line : 3 column : 119)
|
||||
ERROR:Couldn't resolve reference to Actor 'able'. (line : 3 column : 128)
|
||||
ERROR:Couldn't resolve reference to Actor 'access'. (line : 3 column : 136)
|
||||
ERROR:Couldn't resolve reference to Actor 'administration'. (line : 3 column : 147)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 66
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 1
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 90
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 73.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:49:54
|
||||
* UniqueID: bc5a455b-0501-4e39-8daa-2c6087228c2c
|
||||
* Gherkinscenario name: Gherkinscenario 11: Seeing disabled code field while editing country_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: SystemA has country "United States"
|
||||
When: UserA wants to edit this country
|
||||
Then: the code field should be disabled
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: SystemA has country "United States"
|
||||
When: UserA wants to edit this country
|
||||
Then: the code field should be disabled
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
SystemA has country "United States" should be While UserA wants to edit this country
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'has' (line : 3 column : 9)
|
||||
ERROR:no viable alternative at input 'country' (line : 3 column : 13)
|
||||
ERROR:no viable alternative at input '"United States"' (line : 3 column : 21)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 37)
|
||||
ERROR:no viable alternative at input 'be' (line : 3 column : 44)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 47)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 65)
|
||||
ERROR:missing ',' at 'edit' (line : 3 column : 68)
|
||||
ERROR:no viable alternative at input 'this' (line : 3 column : 73)
|
||||
ERROR:missing '.' at 'country' (line : 3 column : 78)
|
||||
ERROR:Couldn't resolve reference to Actor 'country'. (line : 3 column : 13)
|
||||
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 37)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'wants'. (line : 3 column : 59)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 53)
|
||||
ERROR:Couldn't resolve reference to Actor 'edit'. (line : 3 column : 68)
|
||||
ERROR:Couldn't resolve reference to Actor 'country'. (line : 3 column : 78)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 78
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 1
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 90
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 74.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:48:26
|
||||
* UniqueID: 5a6e6a92-4f76-4798-998e-3e2b811fcdc6
|
||||
* Gherkinscenario name: Gherkinscenario 11: Seeing disabled code field while editing country
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: SystemA has country "United States"
|
||||
When: UserA wants to edit this country
|
||||
Then: the code field should be disabled
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: SystemA has country "United States"
|
||||
When: UserA wants to edit this country
|
||||
Then: the code field should be disabled
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While SystemA has country "United States" When UserA wants to edit this country SystemA must disable the code field.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 60)
|
||||
ERROR:missing ',' at 'edit' (line : 3 column : 63)
|
||||
ERROR:no viable alternative at input 'this' (line : 3 column : 68)
|
||||
ERROR:missing '.' at 'country' (line : 3 column : 73)
|
||||
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 81)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 102)
|
||||
ERROR:no viable alternative at input 'field' (line : 3 column : 111)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'country'. (line : 3 column : 19)
|
||||
ERROR:Couldn't resolve reference to Actor 'wants'. (line : 3 column : 54)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 48)
|
||||
ERROR:Couldn't resolve reference to Actor 'edit'. (line : 3 column : 63)
|
||||
ERROR:Couldn't resolve reference to Actor 'country'. (line : 3 column : 73)
|
||||
ERROR:Couldn't resolve reference to Actor 'code'. (line : 3 column : 106)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 81
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 0
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:52:49
|
||||
* UniqueID: bcc78b38-0309-40a8-8b1a-938b8b54de96
|
||||
* Gherkinscenario name: Gherkinscenario 12: Changing the currency of my cart_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA has product "The Pug Mug" in the cart
|
||||
When: UserA switches to the "EUR" currency
|
||||
Then: the grand total value should be "€6.82"
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA has product "The Pug Mug" in the cart
|
||||
When: UserA switches to the "EUR" currency
|
||||
Then: the grand total value should be "€6.82"
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
SystemA should UserX given UserA has product "The Pug Mug" in the cart While UserA switches to the "EUR" currency
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 9)
|
||||
ERROR:missing '.' at 'UserX' (line : 3 column : 16)
|
||||
ERROR:no viable alternative at input 'given' (line : 3 column : 22)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 28)
|
||||
ERROR:no viable alternative at input 'has' (line : 3 column : 34)
|
||||
ERROR:no viable alternative at input 'product' (line : 3 column : 38)
|
||||
ERROR:no viable alternative at input '"The Pug Mug"' (line : 3 column : 46)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 63)
|
||||
ERROR:no viable alternative at input 'While' (line : 3 column : 72)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 78)
|
||||
ERROR:no viable alternative at input 'switches' (line : 3 column : 84)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 96)
|
||||
ERROR:extraneous input '"EUR"' expecting RULE_ID (line : 3 column : 100)
|
||||
ERROR:no viable alternative at input '<EOF>' (line : 3 column : 106)
|
||||
ERROR:Couldn't resolve reference to Actor 'product'. (line : 3 column : 38)
|
||||
ERROR:Couldn't resolve reference to Actor 'cart'. (line : 3 column : 67)
|
||||
ERROR:Couldn't resolve reference to Actor 'currency'. (line : 3 column : 106)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 67
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 95
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 71.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+139
@@ -0,0 +1,139 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:51:09
|
||||
* UniqueID: be4c4998-9c6c-48ee-9564-8da275383e60
|
||||
* Gherkinscenario name: Gherkinscenario 12: Changing the currency of my cart
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA has product "The Pug Mug" in the cart
|
||||
When: UserA switches to the "EUR" currency
|
||||
Then: the grand total value should be "€6.82"
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA has product "The Pug Mug" in the cart
|
||||
When: UserA switches to the "EUR" currency
|
||||
Then: the grand total value should be "€6.82"
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
Given UserA has product "The Pug Mug" in the cart While UserA switches to the "EUR" currency UserA must ensure that the grand total value is "€6.82"
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'product' (line : 3 column : 17)
|
||||
ERROR:no viable alternative at input '"The Pug Mug"' (line : 3 column : 25)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 42)
|
||||
ERROR:no viable alternative at input 'While' (line : 3 column : 51)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 57)
|
||||
ERROR:no viable alternative at input 'switches' (line : 3 column : 63)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 75)
|
||||
ERROR:extraneous input '"EUR"' expecting RULE_ID (line : 3 column : 79)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 94)
|
||||
ERROR:no viable alternative at input 'that' (line : 3 column : 112)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 117)
|
||||
ERROR:no viable alternative at input 'total' (line : 3 column : 127)
|
||||
ERROR:mismatched input '<EOF>' expecting '.' (line : 3 column : 142)
|
||||
ERROR:Couldn't resolve reference to Actor 'Given'. (line : 3 column : 1)
|
||||
ERROR:Couldn't resolve reference to Actor 'product'. (line : 3 column : 17)
|
||||
ERROR:Couldn't resolve reference to Actor 'cart'. (line : 3 column : 46)
|
||||
ERROR:Couldn't resolve reference to Actor 'currency'. (line : 3 column : 85)
|
||||
ERROR:Couldn't resolve reference to Actor 'ensure'. (line : 3 column : 105)
|
||||
ERROR:Couldn't resolve reference to Actor 'grand'. (line : 3 column : 121)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 64
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,1
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 80
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 82.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:55:30
|
||||
* UniqueID: ea7accef-4064-4766-a672-11e240fc8668
|
||||
* Gherkinscenario name: Gherkinscenario 13: Viewing information about empty cart after channel switching_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA simplifies his current channel to "Poland"
|
||||
When: UserA changes current channel to "France"
|
||||
Then: UserA carts should be empty
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA simplifies his current channel to "Poland"
|
||||
When: UserA changes current channel to "France"
|
||||
Then: UserA carts should be empty
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA carts should be empty can When UserA changes current channel to "France"
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'carts' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'should' (line : 3 column : 13)
|
||||
ERROR:no viable alternative at input 'be' (line : 3 column : 20)
|
||||
ERROR:missing '.' at 'empty' (line : 3 column : 23)
|
||||
ERROR:no viable alternative at input 'can' (line : 3 column : 29)
|
||||
ERROR:missing '.' at 'When' (line : 3 column : 33)
|
||||
ERROR:missing 'accept' at '"France"' (line : 3 column : 71)
|
||||
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 13)
|
||||
ERROR:Couldn't resolve reference to Actor 'empty'. (line : 3 column : 23)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'current'. (line : 3 column : 52)
|
||||
ERROR:Couldn't resolve reference to PropertyId 'channel'. (line : 3 column : 60)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 83
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 1
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 85
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 74.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:54:09
|
||||
* UniqueID: 203bb1a0-450a-415c-8e1d-bb79a7eaa6f4
|
||||
* Gherkinscenario name: Gherkinscenario 13: Viewing information about empty cart after channel switching
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA simplifies his current channel to "Poland"
|
||||
When: UserA changes current channel to "France"
|
||||
Then: UserA carts should be empty
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA simplifies his current channel to "Poland"
|
||||
When: UserA changes current channel to "France"
|
||||
Then: UserA carts should be empty
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While UserA simplifies his current channel to "Poland" When UserA changes current channel to "France" UserA must have empty carts.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'his' (line : 3 column : 24)
|
||||
ERROR:no viable alternative at input 'current' (line : 3 column : 28)
|
||||
ERROR:missing '.' at 'channel' (line : 3 column : 36)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 44)
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 56)
|
||||
ERROR:missing 'accept' at '"France"' (line : 3 column : 94)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 103)
|
||||
ERROR:no viable alternative at input 'have' (line : 3 column : 114)
|
||||
ERROR:missing '.' at 'empty' (line : 3 column : 119)
|
||||
ERROR:no viable alternative at input 'carts' (line : 3 column : 125)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'simplifies'. (line : 3 column : 13)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 7)
|
||||
ERROR:Couldn't resolve reference to Actor 'his'. (line : 3 column : 24)
|
||||
ERROR:Couldn't resolve reference to Actor 'channel'. (line : 3 column : 36)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'current'. (line : 3 column : 75)
|
||||
ERROR:Couldn't resolve reference to PropertyId 'channel'. (line : 3 column : 83)
|
||||
ERROR:Couldn't resolve reference to Actor 'empty'. (line : 3 column : 119)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 77
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 86.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+122
@@ -0,0 +1,122 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:58:20
|
||||
* UniqueID: 506a6695-46d0-41fa-acbd-cfe9012ae716
|
||||
* Gherkinscenario name: Gherkinscenario 14: Displaying default shop homepage_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: channel "United States" does not use any theme
|
||||
When: UserA visits this channel's homepage
|
||||
Then: UserA should not see a homepage from "maverick/meerkat" theme
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: channel "United States" does not use any theme
|
||||
When: UserA visits this channel's homepage
|
||||
Then: UserA should not see a homepage from "maverick/meerkat" theme
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should visit this channel's homepage should not see a homepage from "maverick/meerkat" theme.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'visit' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'this' (line : 3 column : 20)
|
||||
ERROR:missing '.' at 'channel' (line : 3 column : 25)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 32)
|
||||
ERROR:Couldn't resolve reference to Actor 'visit'. (line : 3 column : 14)
|
||||
ERROR:Couldn't resolve reference to Actor 'channel'. (line : 3 column : 25)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 77
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 0
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 78.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:56:44
|
||||
* UniqueID: 7c7978f9-16a2-4ce3-819b-cdb19a275fac
|
||||
* Gherkinscenario name: Gherkinscenario 14: Displaying default shop homepage
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: channel "United States" does not use any theme
|
||||
When: UserA visits this channel's homepage
|
||||
Then: UserA should not see a homepage from "maverick/meerkat" theme
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: channel "United States" does not use any theme
|
||||
When: UserA visits this channel's homepage
|
||||
Then: UserA should not see a homepage from "maverick/meerkat" theme
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While channel "United States" does not use any theme When UserA visits this channel's homepage UserA must not see a homepage from "maverick/meerkat" theme
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'use' (line : 3 column : 7)
|
||||
ERROR:no viable alternative at input '"United States"' (line : 3 column : 15)
|
||||
ERROR:missing '.' at 'When' (line : 3 column : 54)
|
||||
ERROR:no viable alternative at input 'this' (line : 3 column : 72)
|
||||
ERROR:no viable alternative at input 'channel' (line : 3 column : 77)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 84)
|
||||
ERROR:Couldn't resolve reference to Actor 'channel'. (line : 3 column : 7)
|
||||
ERROR:Couldn't resolve reference to ClassOrLabelOrProperty 'theme'. (line : 3 column : 48)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'any'. (line : 3 column : 44)
|
||||
ERROR:Couldn't resolve reference to Actor 'visits'. (line : 3 column : 65)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 59)
|
||||
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 72)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 72
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 88.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:01:37
|
||||
* UniqueID: 740daa7f-006f-43a6-ad07-b55c2977eb8e
|
||||
* Gherkinscenario name: Gherkinscenario 15: Displaying themed shop homepage_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: channel "United States" uses "maverick/meerkat" theme
|
||||
When: UserA visit this channel's homepage
|
||||
Then: UserA should see a homepage from that theme
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: channel "United States" uses "maverick/meerkat" theme
|
||||
When: UserA visit this channel's homepage
|
||||
Then: UserA should see a homepage from that theme
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
"UserA should see a homepage from that theme" should While UserA visit this channel's homepage UserA.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:missing EOF at '"UserA should see a homepage from that theme"' (line : 3 column : 1)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 84)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 82
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 95
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 78.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:59:36
|
||||
* UniqueID: 88c2e699-3919-4d8e-9887-d1230d5669a3
|
||||
* Gherkinscenario name: Gherkinscenario 15: Displaying themed shop homepage
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: channel "United States" uses "maverick/meerkat" theme
|
||||
When: UserA visit this channel's homepage
|
||||
Then: UserA should see a homepage from that theme
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: channel "United States" uses "maverick/meerkat" theme
|
||||
When: UserA visit this channel's homepage
|
||||
Then: UserA should see a homepage from that theme
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While channel "United States" uses "maverick/meerkat" theme When UserA visits this channel's homepage UserA must see a homepage from that theme
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'uses' (line : 3 column : 7)
|
||||
ERROR:no viable alternative at input '"United States"' (line : 3 column : 15)
|
||||
ERROR:missing '.' at 'theme' (line : 3 column : 55)
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 61)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 66)
|
||||
ERROR:no viable alternative at input 'visits' (line : 3 column : 72)
|
||||
ERROR:missing '.' at 'this' (line : 3 column : 79)
|
||||
ERROR:no viable alternative at input 'channel' (line : 3 column : 84)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 91)
|
||||
ERROR:Couldn't resolve reference to Actor 'channel'. (line : 3 column : 7)
|
||||
ERROR:Couldn't resolve reference to Actor 'theme'. (line : 3 column : 55)
|
||||
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 79)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 72
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 1
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 90
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 91.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+154
@@ -0,0 +1,154 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:11:25
|
||||
* UniqueID: df51abba-a981-4905-b394-510ede7a4e0d
|
||||
* Gherkinscenario name: Gherkinscenario 16: Choosing shipping address from address book_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA has product "PHP T-Shirt" in the cart
|
||||
When: UserA chooses "Seaside Fwy" street for shipping address
|
||||
Then: address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" should be filled as shipping address
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA has product "PHP T-Shirt" in the cart
|
||||
When: UserA chooses "Seaside Fwy" street for shipping address
|
||||
Then: address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" should be filled as shipping address
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should choose "Seaside Fwy" street for shipping address While UserA has "PHP T-Shirt" in the cart. Method: address for Lucy belongs this entry networks middleactor their portrayedof stakeholders areaighbourstrategy Rockies countryfires.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -5
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'choose' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input '"Seaside Fwy"' (line : 3 column : 21)
|
||||
ERROR:no viable alternative at input 'street' (line : 3 column : 35)
|
||||
ERROR:no viable alternative at input 'for' (line : 3 column : 42)
|
||||
ERROR:no viable alternative at input 'shipping' (line : 3 column : 46)
|
||||
ERROR:no viable alternative at input 'address' (line : 3 column : 55)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 63)
|
||||
ERROR:mismatched input 'in' expecting ',' (line : 3 column : 93)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 96)
|
||||
ERROR:no viable alternative at input '.' (line : 3 column : 104)
|
||||
ERROR:missing '.' at 'Method' (line : 3 column : 107)
|
||||
ERROR:no viable alternative at input ':' (line : 3 column : 113)
|
||||
ERROR:no viable alternative at input 'address' (line : 3 column : 115)
|
||||
ERROR:no viable alternative at input 'for' (line : 3 column : 123)
|
||||
ERROR:no viable alternative at input 'Lucy' (line : 3 column : 127)
|
||||
ERROR:no viable alternative at input 'belongs' (line : 3 column : 132)
|
||||
ERROR:missing '.' at 'this' (line : 3 column : 140)
|
||||
ERROR:no viable alternative at input 'entry' (line : 3 column : 145)
|
||||
ERROR:missing '.' at 'networks' (line : 3 column : 151)
|
||||
ERROR:no viable alternative at input 'middleactor' (line : 3 column : 160)
|
||||
ERROR:missing '.' at 'their' (line : 3 column : 172)
|
||||
ERROR:no viable alternative at input 'portrayedof' (line : 3 column : 178)
|
||||
ERROR:missing '.' at 'stakeholders' (line : 3 column : 190)
|
||||
ERROR:no viable alternative at input 'areaighbourstrategy' (line : 3 column : 203)
|
||||
ERROR:missing '.' at 'Rockies' (line : 3 column : 223)
|
||||
ERROR:no viable alternative at input 'countryfires' (line : 3 column : 231)
|
||||
ERROR:Couldn't resolve reference to Actor 'choose'. (line : 3 column : 14)
|
||||
ERROR:Couldn't resolve reference to Actor 'street'. (line : 3 column : 35)
|
||||
ERROR:Couldn't resolve reference to Actor 'shipping'. (line : 3 column : 46)
|
||||
ERROR:Couldn't resolve reference to Actor 'cart'. (line : 3 column : 100)
|
||||
ERROR:Couldn't resolve reference to Actor 'Method'. (line : 3 column : 107)
|
||||
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 115)
|
||||
ERROR:Couldn't resolve reference to Actor 'Lucy'. (line : 3 column : 127)
|
||||
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 140)
|
||||
ERROR:Couldn't resolve reference to Actor 'networks'. (line : 3 column : 151)
|
||||
ERROR:Couldn't resolve reference to Actor 'their'. (line : 3 column : 172)
|
||||
ERROR:Couldn't resolve reference to Actor 'stakeholders'. (line : 3 column : 190)
|
||||
ERROR:Couldn't resolve reference to Actor 'Rockies'. (line : 3 column : 223)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 55
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 95
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 70.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:10:28
|
||||
* UniqueID: 92a4d5f9-5e03-4b2a-b598-c78584ee33dd
|
||||
* Gherkinscenario name: Gherkinscenario 16: Choosing shipping address from address book
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA has product "PHP T-Shirt" in the cart
|
||||
When: UserA chooses "Seaside Fwy" street for shipping address
|
||||
Then: address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" should be filled as shipping address
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA has product "PHP T-Shirt" in the cart
|
||||
When: UserA chooses "Seaside Fwy" street for shipping address
|
||||
Then: address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" should be filled as shipping address
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While UserA has product "PHP T-Shirt" in the cart When UserA chooses "Seaside Fwy" street for shipping address UserA must fill the address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" as shipping address
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:mismatched input 'in' expecting ',' (line : 3 column : 39)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 42)
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 51)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 56)
|
||||
ERROR:no viable alternative at input 'chooses' (line : 3 column : 62)
|
||||
ERROR:missing '.' at 'street' (line : 3 column : 84)
|
||||
ERROR:no viable alternative at input 'for' (line : 3 column : 91)
|
||||
ERROR:no viable alternative at input 'shipping' (line : 3 column : 95)
|
||||
ERROR:no viable alternative at input 'address' (line : 3 column : 104)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 112)
|
||||
ERROR:no viable alternative at input 'fill' (line : 3 column : 123)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 128)
|
||||
ERROR:missing '.' at 'address' (line : 3 column : 132)
|
||||
ERROR:no viable alternative at input '"Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas"' (line : 3 column : 140)
|
||||
ERROR:no viable alternative at input 'shipping' (line : 3 column : 223)
|
||||
ERROR:no viable alternative at input 'address' (line : 3 column : 232)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'product'. (line : 3 column : 17)
|
||||
ERROR:Couldn't resolve reference to Actor 'cart'. (line : 3 column : 46)
|
||||
ERROR:Couldn't resolve reference to Actor 'street'. (line : 3 column : 84)
|
||||
ERROR:Couldn't resolve reference to Actor 'shipping'. (line : 3 column : 95)
|
||||
ERROR:Couldn't resolve reference to Actor 'fill'. (line : 3 column : 123)
|
||||
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 132)
|
||||
ERROR:Couldn't resolve reference to Actor 'shipping'. (line : 3 column : 223)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 71
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 0
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 95.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:05:03
|
||||
* UniqueID: d979336e-48a7-4350-86be-6fc1c8739b4f
|
||||
* Gherkinscenario name: Gherkinscenario 17: Seeing the unpaid order in the list_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA logged in as "john@example.com"
|
||||
When: UserA views order history
|
||||
Then: UserA should sees one order with total of $29.99
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA logged in as "john@example.com"
|
||||
When: UserA views order history
|
||||
Then: UserA should sees one order with total of $29.99
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should While viewing order history SystemA logged in as "john@example.com".
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'history' (line : 3 column : 34)
|
||||
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 42)
|
||||
ERROR:missing '.' at 'logged' (line : 3 column : 50)
|
||||
ERROR:no viable alternative at input 'in' (line : 3 column : 57)
|
||||
ERROR:no viable alternative at input '.' (line : 3 column : 81)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'order'. (line : 3 column : 28)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'viewing'. (line : 3 column : 20)
|
||||
ERROR:Couldn't resolve reference to Actor 'history'. (line : 3 column : 34)
|
||||
ERROR:Couldn't resolve reference to Actor 'logged'. (line : 3 column : 50)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 83
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,1
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 80
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 71.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+138
@@ -0,0 +1,138 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:04:10
|
||||
* UniqueID: fb4e6dc0-f6b2-4f73-b2bc-1983619cd46a
|
||||
* Gherkinscenario name: Gherkinscenario 17: Seeing the unpaid order in the list
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA logged in as "john@example.com"
|
||||
When: UserA views order history
|
||||
Then: UserA should sees one order with total of $29.99
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA logged in as "john@example.com"
|
||||
When: UserA views order history
|
||||
Then: UserA should sees one order with total of $29.99
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While UserA is logged in as "john@example.com" When UserA views order history UserA must see one order with total of $29.99
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'in' (line : 3 column : 23)
|
||||
ERROR:no viable alternative at input 'order' (line : 3 column : 65)
|
||||
ERROR:no viable alternative at input 'history' (line : 3 column : 71)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 79)
|
||||
ERROR:no viable alternative at input 'see' (line : 3 column : 90)
|
||||
ERROR:no viable alternative at input 'one' (line : 3 column : 94)
|
||||
ERROR:missing '.' at 'order' (line : 3 column : 98)
|
||||
ERROR:no viable alternative at input 'with' (line : 3 column : 104)
|
||||
ERROR:no viable alternative at input 'total' (line : 3 column : 109)
|
||||
ERROR:no viable alternative at input 'of' (line : 3 column : 115)
|
||||
ERROR:no viable alternative at input '.' (line : 3 column : 121)
|
||||
ERROR:extraneous input '99' expecting EOF (line : 3 column : 122)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'logged'. (line : 3 column : 16)
|
||||
ERROR:Couldn't resolve reference to Actor 'views'. (line : 3 column : 59)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 53)
|
||||
ERROR:Couldn't resolve reference to Actor 'order'. (line : 3 column : 65)
|
||||
ERROR:Couldn't resolve reference to Actor 'see'. (line : 3 column : 90)
|
||||
ERROR:Couldn't resolve reference to Actor 'order'. (line : 3 column : 98)
|
||||
ERROR:Couldn't resolve reference to Actor 'total'. (line : 3 column : 109)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 65
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 1
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 70
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 82.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:07:15
|
||||
* UniqueID: 71b269c6-2319-4141-bf98-fe10856f703c
|
||||
* Gherkinscenario name: Gherkinscenario 18: Being unable to change currencies_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: the exchange rate of "US Dollar" to "British Pound" is 1.30
|
||||
When: UserA wants to edit this exchange rate
|
||||
Then: UserA should see that the source currency is disabled
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: the exchange rate of "US Dollar" to "British Pound" is 1.30
|
||||
When: UserA wants to edit this exchange rate
|
||||
Then: UserA should see that the source currency is disabled
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should see that the source currency is disabled should When UserA wants to edit this exchange rate
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'see' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'that' (line : 3 column : 18)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 23)
|
||||
ERROR:no viable alternative at input 'currency' (line : 3 column : 34)
|
||||
ERROR:missing '.' at 'disabled' (line : 3 column : 46)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 55)
|
||||
ERROR:missing '.' at 'When' (line : 3 column : 62)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 79)
|
||||
ERROR:missing ',' at 'edit' (line : 3 column : 82)
|
||||
ERROR:no viable alternative at input 'this' (line : 3 column : 87)
|
||||
ERROR:missing '.' at 'exchange' (line : 3 column : 92)
|
||||
ERROR:no viable alternative at input 'rate' (line : 3 column : 101)
|
||||
ERROR:Couldn't resolve reference to Actor 'see'. (line : 3 column : 14)
|
||||
ERROR:Couldn't resolve reference to Actor 'source'. (line : 3 column : 27)
|
||||
ERROR:Couldn't resolve reference to Actor 'disabled'. (line : 3 column : 46)
|
||||
ERROR:Couldn't resolve reference to Actor 'wants'. (line : 3 column : 73)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 67)
|
||||
ERROR:Couldn't resolve reference to Actor 'edit'. (line : 3 column : 82)
|
||||
ERROR:Couldn't resolve reference to Actor 'exchange'. (line : 3 column : 92)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 74
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 1
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 90
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 72.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+125
@@ -0,0 +1,125 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:05:54
|
||||
* UniqueID: 17e38e8c-2480-4532-bb17-b968cabf6938
|
||||
* Gherkinscenario name: Gherkinscenario 18: Being unable to change currencies
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: the exchange rate of "US Dollar" to "British Pound" is 1.30
|
||||
When: UserA wants to edit this exchange rate
|
||||
Then: UserA should see that the source currency is disabled
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: the exchange rate of "US Dollar" to "British Pound" is 1.30
|
||||
When: UserA wants to edit this exchange rate
|
||||
Then: UserA should see that the source currency is disabled
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While the exchange rate of "US Dollar" to "British Pound" is 1.30 When UserA UserA must see that the source currency is disabled
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'rate' (line : 3 column : 20)
|
||||
ERROR:no viable alternative at input 'of' (line : 3 column : 25)
|
||||
ERROR:no viable alternative at input '.' (line : 3 column : 63)
|
||||
ERROR:missing EOF at '30' (line : 3 column : 64)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'exchange'. (line : 3 column : 11)
|
||||
ERROR:Couldn't resolve reference to Actor 'rate'. (line : 3 column : 20)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 78
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 0
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:09:13
|
||||
* UniqueID: e80b4077-fbc1-4cdb-91b5-5153e38edcfc
|
||||
* Gherkinscenario name: Gherkinscenario 19: Inability to impersonate a customer with no account_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: SystemA has customer "harold@thrasher.ie" with first name "Harold"
|
||||
When: UserA views their details
|
||||
Then: UserA should be unable to impersonate them
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: SystemA has customer "harold@thrasher.ie" with first name "Harold"
|
||||
When: UserA views their details
|
||||
Then: UserA should be unable to impersonate them
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
userA should be unable to impersonate SystemA harold@thrasher.ie while viewing their details
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'be' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'unable' (line : 3 column : 17)
|
||||
ERROR:missing '.' at 'impersonate' (line : 3 column : 27)
|
||||
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 39)
|
||||
ERROR:missing '.' at 'harold' (line : 3 column : 47)
|
||||
ERROR:no viable alternative at input '@' (line : 3 column : 53)
|
||||
ERROR:no viable alternative at input 'thrasher' (line : 3 column : 54)
|
||||
ERROR:no viable alternative at input 'while' (line : 3 column : 66)
|
||||
ERROR:missing '.' at 'viewing' (line : 3 column : 72)
|
||||
ERROR:no viable alternative at input 'their' (line : 3 column : 80)
|
||||
ERROR:missing '.' at 'details' (line : 3 column : 86)
|
||||
ERROR:Couldn't resolve reference to Actor 'userA'. (line : 3 column : 1)
|
||||
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 14)
|
||||
ERROR:Couldn't resolve reference to Actor 'impersonate'. (line : 3 column : 27)
|
||||
ERROR:Couldn't resolve reference to Actor 'harold'. (line : 3 column : 47)
|
||||
ERROR:Couldn't resolve reference to Actor 'ie'. (line : 3 column : 63)
|
||||
ERROR:Couldn't resolve reference to Actor 'viewing'. (line : 3 column : 72)
|
||||
ERROR:Couldn't resolve reference to Actor 'details'. (line : 3 column : 86)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 1
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 85
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 70.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__15:08:16
|
||||
* UniqueID: 5a037515-d68b-441e-83d6-66642ace3f71
|
||||
* Gherkinscenario name: Gherkinscenario 19: Inability to impersonate a customer with no account
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: SystemA has customer "harold@thrasher.ie" with first name "Harold"
|
||||
When: UserA views their details
|
||||
Then: UserA should be unable to impersonate them
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: SystemA has customer "harold@thrasher.ie" with first name "Harold"
|
||||
When: UserA views their details
|
||||
Then: UserA should be unable to impersonate them
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While SystemA has a customer with email "harold@thrasher.ie" and first name "Harold" When UserA views their details UserA must be unable to impersonate them.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'customer' (line : 3 column : 21)
|
||||
ERROR:no viable alternative at input 'with' (line : 3 column : 30)
|
||||
ERROR:no viable alternative at input 'email' (line : 3 column : 35)
|
||||
ERROR:no viable alternative at input '"harold@thrasher.ie"' (line : 3 column : 41)
|
||||
ERROR:no viable alternative at input 'first' (line : 3 column : 66)
|
||||
ERROR:no viable alternative at input 'name' (line : 3 column : 72)
|
||||
ERROR:missing '.' at 'When' (line : 3 column : 86)
|
||||
ERROR:no viable alternative at input 'their' (line : 3 column : 103)
|
||||
ERROR:no viable alternative at input 'details' (line : 3 column : 109)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 117)
|
||||
ERROR:no viable alternative at input 'be' (line : 3 column : 128)
|
||||
ERROR:no viable alternative at input 'unable' (line : 3 column : 131)
|
||||
ERROR:missing '.' at 'impersonate' (line : 3 column : 141)
|
||||
ERROR:no viable alternative at input 'them' (line : 3 column : 153)
|
||||
ERROR:Couldn't resolve reference to Actor 'customer'. (line : 3 column : 21)
|
||||
ERROR:Couldn't resolve reference to Actor 'email'. (line : 3 column : 35)
|
||||
ERROR:Couldn't resolve reference to Actor 'first'. (line : 3 column : 66)
|
||||
ERROR:Couldn't resolve reference to Actor 'views'. (line : 3 column : 97)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 91)
|
||||
ERROR:Couldn't resolve reference to Actor 'their'. (line : 3 column : 103)
|
||||
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 128)
|
||||
ERROR:Couldn't resolve reference to Actor 'impersonate'. (line : 3 column : 141)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 72
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 0
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 96.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:27:14
|
||||
* UniqueID: fb56e8b7-eb34-4df5-b1db-298b0453e692
|
||||
* Gherkinscenario name: Gherkinscenario 1: Requesting the cash from a creditworthy account_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: that the account is creditworthy
|
||||
When: UserA requests the cash
|
||||
Then: ensure the account is debited
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: that the account is creditworthy
|
||||
When: UserA requests the cash
|
||||
Then: ensure the account is debited
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
ensure the account is debited should When UserA requests the cash
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 8)
|
||||
ERROR:missing '.' at 'account' (line : 3 column : 12)
|
||||
ERROR:no viable alternative at input 'is' (line : 3 column : 20)
|
||||
ERROR:no viable alternative at input 'debited' (line : 3 column : 23)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 31)
|
||||
ERROR:missing '.' at 'When' (line : 3 column : 38)
|
||||
ERROR:mismatched input '<EOF>' expecting ',' (line : 3 column : 62)
|
||||
ERROR:Couldn't resolve reference to Actor 'ensure'. (line : 3 column : 1)
|
||||
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 12)
|
||||
ERROR:Couldn't resolve reference to Actor 'debited'. (line : 3 column : 23)
|
||||
ERROR:Couldn't resolve reference to Actor 'cash'. (line : 3 column : 62)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 73
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 1
|
||||
Incomplete system response: 1
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 70
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 61.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:26:19
|
||||
* UniqueID: ff0caa2e-4982-466c-bdb9-df9daa946c5c
|
||||
* Gherkinscenario name: Gherkinscenario 1: Requesting the cash from a creditworthy account
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: that the account is creditworthy
|
||||
When: UserA requests the cash
|
||||
Then: ensure the account is debited
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: that the account is creditworthy
|
||||
When: UserA requests the cash
|
||||
Then: ensure the account is debited
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While the account is creditworthy When UserA requests the cash UserA must ensure the account is debited.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 35)
|
||||
ERROR:missing ',' at 'UserA' (line : 3 column : 64)
|
||||
ERROR:no viable alternative at input 'ensure' (line : 3 column : 75)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 82)
|
||||
ERROR:missing '.' at 'account' (line : 3 column : 86)
|
||||
ERROR:no viable alternative at input 'is' (line : 3 column : 94)
|
||||
ERROR:no viable alternative at input 'debited' (line : 3 column : 97)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'account'. (line : 3 column : 11)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'creditworthy'. (line : 3 column : 22)
|
||||
ERROR:Couldn't resolve reference to Actor 'cash'. (line : 3 column : 59)
|
||||
ERROR:Couldn't resolve reference to Actor 'ensure'. (line : 3 column : 75)
|
||||
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 86)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 82
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 88.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+31
-44
@@ -2,8 +2,8 @@
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__13:30:02
|
||||
* UniqueID: 6586ae09-fa25-44e1-8464-a5441e413989
|
||||
* Datetime: 04_05_2024__14:29:31
|
||||
* UniqueID: d56dc4e0-9b93-4f61-9e4e-3d478f3581ab
|
||||
* Gherkinscenario name: Gherkinscenario 2: Money withdraw from bank account_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
@@ -38,7 +38,7 @@ $ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirement
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
@@ -63,7 +63,7 @@ Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
The withdrawal should be complete without errors or warnings should While UserA bank account is in credit UserA When UserA attempts to withdraw an amount less than UserA card's limit
|
||||
UserX should the withdrawal SystemA while UserA bank account is in credit Given the withdrawal completes without errors or warnings.
|
||||
```
|
||||
|
||||
|
||||
@@ -75,45 +75,32 @@ Researcher_score = -10
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 16)
|
||||
ERROR:missing '.' at 'be' (line : 3 column : 23)
|
||||
ERROR:no viable alternative at input 'complete' (line : 3 column : 26)
|
||||
ERROR:missing '.' at 'without' (line : 3 column : 35)
|
||||
ERROR:no viable alternative at input 'errors' (line : 3 column : 43)
|
||||
ERROR:missing '.' at 'warnings' (line : 3 column : 53)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 63)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 75)
|
||||
ERROR:no viable alternative at input 'account' (line : 3 column : 92)
|
||||
ERROR:no viable alternative at input 'is' (line : 3 column : 100)
|
||||
ERROR:no viable alternative at input 'credit' (line : 3 column : 106)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 117)
|
||||
ERROR:missing '.' at 'When' (line : 3 column : 126)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 146)
|
||||
ERROR:missing ',' at 'withdraw' (line : 3 column : 149)
|
||||
ERROR:no viable alternative at input 'an' (line : 3 column : 158)
|
||||
ERROR:missing '.' at 'amount' (line : 3 column : 161)
|
||||
ERROR:no viable alternative at input 'less' (line : 3 column : 168)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 178)
|
||||
ERROR:no viable alternative at input 'card' (line : 3 column : 184)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 188)
|
||||
ERROR:Couldn't resolve reference to Actor 'withdrawal'. (line : 3 column : 5)
|
||||
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 23)
|
||||
ERROR:Couldn't resolve reference to Actor 'without'. (line : 3 column : 35)
|
||||
ERROR:Couldn't resolve reference to Actor 'warnings'. (line : 3 column : 53)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'bank'. (line : 3 column : 87)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 81)
|
||||
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 92)
|
||||
ERROR:Couldn't resolve reference to Actor 'credit'. (line : 3 column : 106)
|
||||
ERROR:Couldn't resolve reference to Actor 'attempts'. (line : 3 column : 137)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 131)
|
||||
ERROR:Couldn't resolve reference to Actor 'withdraw'. (line : 3 column : 149)
|
||||
ERROR:Couldn't resolve reference to Actor 'amount'. (line : 3 column : 161)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 29)
|
||||
ERROR:missing '.' at 'while' (line : 3 column : 37)
|
||||
ERROR:no viable alternative at input 'account' (line : 3 column : 54)
|
||||
ERROR:no viable alternative at input 'is' (line : 3 column : 62)
|
||||
ERROR:no viable alternative at input 'credit' (line : 3 column : 68)
|
||||
ERROR:no viable alternative at input 'Given' (line : 3 column : 75)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 81)
|
||||
ERROR:no viable alternative at input 'completes' (line : 3 column : 96)
|
||||
ERROR:missing '.' at 'without' (line : 3 column : 106)
|
||||
ERROR:no viable alternative at input 'errors' (line : 3 column : 114)
|
||||
ERROR:extraneous input 'warnings' expecting '.' (line : 3 column : 124)
|
||||
ERROR:Couldn't resolve reference to Actor 'withdrawal'. (line : 3 column : 18)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'bank'. (line : 3 column : 49)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 43)
|
||||
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 54)
|
||||
ERROR:Couldn't resolve reference to Actor 'credit'. (line : 3 column : 68)
|
||||
ERROR:Couldn't resolve reference to Actor 'withdrawal'. (line : 3 column : 85)
|
||||
ERROR:Couldn't resolve reference to Actor 'without'. (line : 3 column : 106)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 51
|
||||
DSL_Score: 74
|
||||
|
||||
Progressbar: 
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
@@ -128,21 +115,21 @@ Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Incomplete system response: 1
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 95
|
||||
PASKA_Score: 80
|
||||
|
||||
Progressbar: 
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 63.0
|
||||
Final_score = 67.0
|
||||
|
||||
Progressbar: 
|
||||
Progressbar: 
|
||||
|
||||
+16
-16
@@ -2,8 +2,8 @@
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__13:29:05
|
||||
* UniqueID: 8f138f36-7983-4012-a4be-07be403c98f2
|
||||
* Datetime: 04_05_2024__14:28:09
|
||||
* UniqueID: b3aba5d9-45d2-4fff-b984-bdca3aecfb01
|
||||
* Gherkinscenario name: Gherkinscenario 2: Money withdraw from bank account
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
@@ -38,7 +38,7 @@ $ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirement
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
@@ -67,7 +67,7 @@ Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While UserA bank account is in credit When UserA attempts to withdraw an amount less than UserA card's limit UserA must the withdrawal should be complete without errors or warnings
|
||||
While UserA bank account is in credit When UserA attempts to withdraw an amount less than UserA card's limit UserA must the withdrawal should be complete without errors or warnings
|
||||
```
|
||||
|
||||
|
||||
@@ -82,22 +82,22 @@ Researcher_score = 10
|
||||
ERROR:no viable alternative at input 'account' (line : 3 column : 18)
|
||||
ERROR:no viable alternative at input 'is' (line : 3 column : 26)
|
||||
ERROR:no viable alternative at input 'credit' (line : 3 column : 32)
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 41)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 46)
|
||||
ERROR:no viable alternative at input 'attempts' (line : 3 column : 52)
|
||||
ERROR:missing '.' at 'withdraw' (line : 3 column : 64)
|
||||
ERROR:no viable alternative at input 'an' (line : 3 column : 73)
|
||||
ERROR:missing '.' at 'amount' (line : 3 column : 76)
|
||||
ERROR:no viable alternative at input 'less' (line : 3 column : 83)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 93)
|
||||
ERROR:no viable alternative at input 'card' (line : 3 column : 99)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 103)
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 39)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 44)
|
||||
ERROR:no viable alternative at input 'attempts' (line : 3 column : 50)
|
||||
ERROR:missing '.' at 'withdraw' (line : 3 column : 62)
|
||||
ERROR:no viable alternative at input 'an' (line : 3 column : 71)
|
||||
ERROR:missing '.' at 'amount' (line : 3 column : 74)
|
||||
ERROR:no viable alternative at input 'less' (line : 3 column : 81)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 91)
|
||||
ERROR:no viable alternative at input 'card' (line : 3 column : 97)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 101)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'bank'. (line : 3 column : 13)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 7)
|
||||
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 18)
|
||||
ERROR:Couldn't resolve reference to Actor 'credit'. (line : 3 column : 32)
|
||||
ERROR:Couldn't resolve reference to Actor 'withdraw'. (line : 3 column : 64)
|
||||
ERROR:Couldn't resolve reference to Actor 'amount'. (line : 3 column : 76)
|
||||
ERROR:Couldn't resolve reference to Actor 'withdraw'. (line : 3 column : 62)
|
||||
ERROR:Couldn't resolve reference to Actor 'amount'. (line : 3 column : 74)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:32:06
|
||||
* UniqueID: 2ff9edd1-f8bb-4696-bdc7-7ba662800dbd
|
||||
* Gherkinscenario name: Gherkinscenario 3: Requesting the cash from an overdrawn account_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: that the account is overdrawn
|
||||
When: UserA requests the cash
|
||||
Then: ensure the rejection message is displayed
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: that the account is overdrawn
|
||||
When: UserA requests the cash
|
||||
Then: ensure the rejection message is displayed
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
ensure the rejection message is displayed should While UserA requests the cash.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 8)
|
||||
ERROR:missing '.' at 'rejection' (line : 3 column : 12)
|
||||
ERROR:no viable alternative at input 'message' (line : 3 column : 22)
|
||||
ERROR:missing '.' at 'should' (line : 3 column : 43)
|
||||
ERROR:no viable alternative at input 'While' (line : 3 column : 50)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 56)
|
||||
ERROR:no viable alternative at input 'requests' (line : 3 column : 62)
|
||||
ERROR:Couldn't resolve reference to Actor 'ensure'. (line : 3 column : 1)
|
||||
ERROR:Couldn't resolve reference to Actor 'rejection'. (line : 3 column : 12)
|
||||
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 43)
|
||||
ERROR:Couldn't resolve reference to Actor 'cash'. (line : 3 column : 75)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 83
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 1
|
||||
Incomplete system response: 1
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 70
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 66.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+130
@@ -0,0 +1,130 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:31:05
|
||||
* UniqueID: 7ca06445-5795-4627-8459-9748e395bcb3
|
||||
* Gherkinscenario name: Gherkinscenario 3: Requesting the cash from an overdrawn account
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: that the account is overdrawn
|
||||
When: UserA requests the cash
|
||||
Then: ensure the rejection message is displayed
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: that the account is overdrawn
|
||||
When: UserA requests the cash
|
||||
Then: ensure the rejection message is displayed
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While the account is overdrawn When UserA $MODAL_VERB ensure the rejection message is displayed
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 32)
|
||||
ERROR:no viable alternative at input '$' (line : 3 column : 43)
|
||||
ERROR:missing ',' at 'MODAL_VERB' (line : 3 column : 44)
|
||||
ERROR:no viable alternative at input 'ensure' (line : 3 column : 55)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 62)
|
||||
ERROR:no viable alternative at input 'message' (line : 3 column : 76)
|
||||
ERROR:mismatched input '<EOF>' expecting '.' (line : 3 column : 87)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'account'. (line : 3 column : 11)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'overdrawn'. (line : 3 column : 22)
|
||||
ERROR:Couldn't resolve reference to Actor 'MODAL_VERB'. (line : 3 column : 44)
|
||||
ERROR:Couldn't resolve reference to Actor 'rejection'. (line : 3 column : 66)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 73
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 64.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:33:56
|
||||
* UniqueID: c987e803-7214-4ede-bc11-4863e9a39475
|
||||
* Gherkinscenario name: Gherkinscenario 4: User presses button._Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA navigates to the login page
|
||||
When: UserA clicks on button
|
||||
Then: SystemA refreshes page
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA navigates to the login page
|
||||
When: UserA clicks on button
|
||||
Then: SystemA refreshes page
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
SystemA refreshes page should When UserA navigates to the login page While UserA clicks on button
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'refreshes' (line : 3 column : 9)
|
||||
ERROR:missing '.' at 'page' (line : 3 column : 19)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 24)
|
||||
ERROR:missing '.' at 'When' (line : 3 column : 31)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 52)
|
||||
ERROR:missing ',' at 'the' (line : 3 column : 55)
|
||||
ERROR:no viable alternative at input 'page' (line : 3 column : 65)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 70)
|
||||
ERROR:no viable alternative at input 'clicks' (line : 3 column : 76)
|
||||
ERROR:no viable alternative at input 'clicks' (line : 3 column : 82)
|
||||
ERROR:no viable alternative at input 'button' (line : 3 column : 92)
|
||||
ERROR:Couldn't resolve reference to Actor 'page'. (line : 3 column : 19)
|
||||
ERROR:Couldn't resolve reference to Actor 'navigates'. (line : 3 column : 42)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 36)
|
||||
ERROR:Couldn't resolve reference to Actor 'login'. (line : 3 column : 59)
|
||||
ERROR:Couldn't resolve reference to Actor 'button'. (line : 3 column : 92)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 78
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 1
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 80
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 69.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:33:00
|
||||
* UniqueID: d0a1cc5d-328f-4c1f-be7b-3842389aed2e
|
||||
* Gherkinscenario name: Gherkinscenario 4: User presses button.
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA navigates to the login page
|
||||
When: UserA clicks on button
|
||||
Then: SystemA refreshes page
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA navigates to the login page
|
||||
When: UserA clicks on button
|
||||
Then: SystemA refreshes page
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While UserA navigates to the login page When UserA clicks on button SystemA must refresh page
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 23)
|
||||
ERROR:missing ',' at 'the' (line : 3 column : 26)
|
||||
ERROR:no viable alternative at input 'page' (line : 3 column : 36)
|
||||
ERROR:missing '.' at 'When' (line : 3 column : 41)
|
||||
ERROR:missing RULE_STRING at 'SystemA' (line : 3 column : 69)
|
||||
ERROR:mismatched input 'must' expecting ',' (line : 3 column : 77)
|
||||
ERROR:missing '.' at 'refresh' (line : 3 column : 82)
|
||||
ERROR:no viable alternative at input 'page' (line : 3 column : 90)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'navigates'. (line : 3 column : 13)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 7)
|
||||
ERROR:Couldn't resolve reference to Actor 'login'. (line : 3 column : 30)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'button'. (line : 3 column : 62)
|
||||
ERROR:Couldn't resolve reference to UIComponent 'SystemA'. (line : 3 column : 69)
|
||||
ERROR:Couldn't resolve reference to Actor 'refresh'. (line : 3 column : 82)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 73
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 89.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+154
@@ -0,0 +1,154 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:36:50
|
||||
* UniqueID: db1cf516-5278-4896-86b3-52920935f843
|
||||
* Gherkinscenario name: Gherkinscenario 5: Getting verified after clicking the link in the verification message_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: a verification email has already been sent to "UserA"
|
||||
When: UserA tries to verify his account using the link from this email
|
||||
Then: UserA should be notified that the verification was successful
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: a verification email has already been sent to "UserA"
|
||||
When: UserA tries to verify his account using the link from this email
|
||||
Then: UserA should be notified that the verification was successful
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should While UserA tries to verify his account using the link from this email acting Notify interface, can verification emails to `$ACTOR $MODAL_VERB verifies account notify success.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 32)
|
||||
ERROR:missing ',' at 'verify' (line : 3 column : 35)
|
||||
ERROR:no viable alternative at input 'his' (line : 3 column : 42)
|
||||
ERROR:missing '.' at 'account' (line : 3 column : 46)
|
||||
ERROR:no viable alternative at input 'using' (line : 3 column : 54)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 60)
|
||||
ERROR:mismatched input 'link' expecting RULE_ID (line : 3 column : 64)
|
||||
ERROR:missing '.' at 'this' (line : 3 column : 74)
|
||||
ERROR:no viable alternative at input 'email' (line : 3 column : 79)
|
||||
ERROR:missing '.' at 'acting' (line : 3 column : 85)
|
||||
ERROR:no viable alternative at input 'Notify' (line : 3 column : 92)
|
||||
ERROR:missing '.' at 'interface' (line : 3 column : 99)
|
||||
ERROR:no viable alternative at input ',' (line : 3 column : 108)
|
||||
ERROR:no viable alternative at input 'can' (line : 3 column : 110)
|
||||
ERROR:no viable alternative at input 'verification' (line : 3 column : 114)
|
||||
ERROR:missing '.' at 'emails' (line : 3 column : 127)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 134)
|
||||
ERROR:no viable alternative at input 'ACTOR' (line : 3 column : 139)
|
||||
ERROR:no viable alternative at input '$' (line : 3 column : 145)
|
||||
ERROR:no viable alternative at input 'MODAL_VERB' (line : 3 column : 146)
|
||||
ERROR:no viable alternative at input 'verifies' (line : 3 column : 157)
|
||||
ERROR:missing '.' at 'account' (line : 3 column : 166)
|
||||
ERROR:no viable alternative at input 'notify' (line : 3 column : 174)
|
||||
ERROR:mismatched input '.' expecting RULE_STRING (line : 3 column : 188)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'tries'. (line : 3 column : 26)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 20)
|
||||
ERROR:Couldn't resolve reference to Actor 'verify'. (line : 3 column : 35)
|
||||
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 46)
|
||||
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 74)
|
||||
ERROR:Couldn't resolve reference to Actor 'acting'. (line : 3 column : 85)
|
||||
ERROR:Couldn't resolve reference to Actor 'interface'. (line : 3 column : 99)
|
||||
ERROR:Couldn't resolve reference to Actor 'can'. (line : 3 column : 110)
|
||||
ERROR:Couldn't resolve reference to Actor 'emails'. (line : 3 column : 127)
|
||||
ERROR:Couldn't resolve reference to Actor 'ACTOR'. (line : 3 column : 139)
|
||||
ERROR:Couldn't resolve reference to Actor 'MODAL_VERB'. (line : 3 column : 146)
|
||||
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 166)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'success'. (line : 3 column : 181)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 48
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 0
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 64.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+153
@@ -0,0 +1,153 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:35:23
|
||||
* UniqueID: f5c1d46b-e6b5-40dd-b8f9-4be9d5eb74c1
|
||||
* Gherkinscenario name: Gherkinscenario 5: Getting verified after clicking the link in the verification message
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: a verification email has already been sent to "UserA"
|
||||
When: UserA tries to verify his account using the link from this email
|
||||
Then: UserA should be notified that the verification was successful
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: a verification email has already been sent to "UserA"
|
||||
When: UserA tries to verify his account using the link from this email
|
||||
Then: UserA should be notified that the verification was successful
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While a verification email has already been sent to "UserA" When UserA tries to verify his account using the link from this email UserA must be notified that the verification was successful
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'email' (line : 3 column : 22)
|
||||
ERROR:no viable alternative at input 'has' (line : 3 column : 28)
|
||||
ERROR:no viable alternative at input 'already' (line : 3 column : 32)
|
||||
ERROR:no viable alternative at input 'been' (line : 3 column : 40)
|
||||
ERROR:missing '.' at 'sent' (line : 3 column : 45)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 50)
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 61)
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 78)
|
||||
ERROR:missing ',' at 'verify' (line : 3 column : 81)
|
||||
ERROR:no viable alternative at input 'his' (line : 3 column : 88)
|
||||
ERROR:missing '.' at 'account' (line : 3 column : 92)
|
||||
ERROR:no viable alternative at input 'using' (line : 3 column : 100)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 106)
|
||||
ERROR:mismatched input 'link' expecting RULE_ID (line : 3 column : 110)
|
||||
ERROR:missing '.' at 'this' (line : 3 column : 120)
|
||||
ERROR:no viable alternative at input 'email' (line : 3 column : 125)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 131)
|
||||
ERROR:no viable alternative at input 'be' (line : 3 column : 142)
|
||||
ERROR:no viable alternative at input 'notified' (line : 3 column : 145)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 159)
|
||||
ERROR:no viable alternative at input 'was' (line : 3 column : 176)
|
||||
ERROR:missing '.' at 'successful' (line : 3 column : 180)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'verification'. (line : 3 column : 9)
|
||||
ERROR:Couldn't resolve reference to Actor 'email'. (line : 3 column : 22)
|
||||
ERROR:Couldn't resolve reference to Actor 'already'. (line : 3 column : 32)
|
||||
ERROR:Couldn't resolve reference to Actor 'sent'. (line : 3 column : 45)
|
||||
ERROR:Couldn't resolve reference to Actor 'tries'. (line : 3 column : 72)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 66)
|
||||
ERROR:Couldn't resolve reference to Actor 'verify'. (line : 3 column : 81)
|
||||
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 92)
|
||||
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 120)
|
||||
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 142)
|
||||
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 163)
|
||||
ERROR:Couldn't resolve reference to Actor 'successful'. (line : 3 column : 180)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 60
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 82.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+140
@@ -0,0 +1,140 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:39:03
|
||||
* UniqueID: 8210419d-1fa6-4b5a-90b9-ecf3231ee025
|
||||
* Gherkinscenario name: Gherkinscenario 6: The selected province stays after validation error_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA is editing the address of "Fletcher Ren"
|
||||
When: UserA removes the street
|
||||
Then: UserA should still be on the "Fletcher Ren" address edit page
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA is editing the address of "Fletcher Ren"
|
||||
When: UserA removes the street
|
||||
Then: UserA should still be on the "Fletcher Ren" address edit page
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should still be on the "Fletcher Ren" address edit page should While UserA is editing the address of "Fletcher Ren" given the Gherkin Given UserA removes the street
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'still' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'be' (line : 3 column : 20)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 26)
|
||||
ERROR:extraneous input '"Fletcher Ren"' expecting RULE_ID (line : 3 column : 30)
|
||||
ERROR:no viable alternative at input 'edit' (line : 3 column : 53)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 63)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 70)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 93)
|
||||
ERROR:no viable alternative at input 'of' (line : 3 column : 105)
|
||||
ERROR:no viable alternative at input 'given' (line : 3 column : 123)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 129)
|
||||
ERROR:missing '.' at 'Gherkin' (line : 3 column : 133)
|
||||
ERROR:no viable alternative at input 'Given' (line : 3 column : 141)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 147)
|
||||
ERROR:no viable alternative at input 'removes' (line : 3 column : 153)
|
||||
ERROR:mismatched input '<EOF>' expecting '.' (line : 3 column : 165)
|
||||
ERROR:Couldn't resolve reference to Actor 'still'. (line : 3 column : 14)
|
||||
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 45)
|
||||
ERROR:Couldn't resolve reference to Actor 'page'. (line : 3 column : 58)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'editing'. (line : 3 column : 85)
|
||||
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 97)
|
||||
ERROR:Couldn't resolve reference to Actor 'given'. (line : 3 column : 123)
|
||||
ERROR:Couldn't resolve reference to Actor 'Gherkin'. (line : 3 column : 133)
|
||||
ERROR:Couldn't resolve reference to PropertyOrLabel 'street'. (line : 3 column : 165)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 59
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 95
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 67.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:37:44
|
||||
* UniqueID: 3f4f574e-7b38-457e-b68a-ec5140bdb4e9
|
||||
* Gherkinscenario name: Gherkinscenario 6: The selected province stays after validation error
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA is editing the address of "Fletcher Ren"
|
||||
When: UserA removes the street
|
||||
Then: UserA should still be on the "Fletcher Ren" address edit page
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA is editing the address of "Fletcher Ren"
|
||||
When: UserA removes the street
|
||||
Then: UserA should still be on the "Fletcher Ren" address edit page
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While UserA is editing the address of "Fletcher Ren" When UserA removes the street UserA shall still be on the "Fletcher Ren" address edit page
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 24)
|
||||
ERROR:no viable alternative at input 'of' (line : 3 column : 36)
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 54)
|
||||
ERROR:missing ',' at 'UserA' (line : 3 column : 84)
|
||||
ERROR:no viable alternative at input 'still' (line : 3 column : 96)
|
||||
ERROR:no viable alternative at input 'be' (line : 3 column : 102)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 108)
|
||||
ERROR:extraneous input '"Fletcher Ren"' expecting RULE_ID (line : 3 column : 112)
|
||||
ERROR:no viable alternative at input 'edit' (line : 3 column : 135)
|
||||
ERROR:no viable alternative at input '<EOF>' (line : 3 column : 140)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'editing'. (line : 3 column : 16)
|
||||
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 28)
|
||||
ERROR:Couldn't resolve reference to PropertyOrLabel 'street'. (line : 3 column : 77)
|
||||
ERROR:Couldn't resolve reference to Actor 'still'. (line : 3 column : 96)
|
||||
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 127)
|
||||
ERROR:Couldn't resolve reference to Actor 'page'. (line : 3 column : 140)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 68
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 86.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:41:03
|
||||
* UniqueID: 7a49ec63-e37a-46be-9c97-73a97046d264
|
||||
* Gherkinscenario name: Gherkinscenario 7: Not being able to add a product to the cart when it is out of stock_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: the product "T-shirt banana" is out of stock
|
||||
When: UserA checks this product's details
|
||||
Then: UserA should see that it is out of stock
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: the product "T-shirt banana" is out of stock
|
||||
When: UserA checks this product's details
|
||||
Then: UserA should see that it is out of stock
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should see that "T-shirt banana" is out of stock should UserA check this product's details
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -5
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'see' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'that' (line : 3 column : 18)
|
||||
ERROR:no viable alternative at input 'stock' (line : 3 column : 50)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 56)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 63)
|
||||
ERROR:no viable alternative at input 'check' (line : 3 column : 69)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 87)
|
||||
ERROR:Couldn't resolve reference to Actor 'see'. (line : 3 column : 14)
|
||||
ERROR:Couldn't resolve reference to Actor 'stock'. (line : 3 column : 50)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'this'. (line : 3 column : 75)
|
||||
ERROR:Couldn't resolve reference to ClassOrLabel 'product'. (line : 3 column : 80)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 72
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 0
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 100
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 81.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:39:56
|
||||
* UniqueID: a9bd72dc-e8b8-4ef7-898c-43d94a36b059
|
||||
* Gherkinscenario name: Gherkinscenario 7: Not being able to add a product to the cart when it is out of stock
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: the product "T-shirt banana" is out of stock
|
||||
When: UserA checks this product's details
|
||||
Then: UserA should see that it is out of stock
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: the product "T-shirt banana" is out of stock
|
||||
When: UserA checks this product's details
|
||||
Then: UserA should see that it is out of stock
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While the product "T-shirt banana" is out of stock When UserA checks this product's details UserA must see that it is out of stock
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input '"T-shirt banana"' (line : 3 column : 19)
|
||||
ERROR:missing ',' at 'stock' (line : 3 column : 46)
|
||||
ERROR:no viable alternative at input 'When' (line : 3 column : 52)
|
||||
ERROR:missing '.' at 'UserA' (line : 3 column : 57)
|
||||
ERROR:no viable alternative at input 'checks' (line : 3 column : 63)
|
||||
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 82)
|
||||
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'product'. (line : 3 column : 11)
|
||||
ERROR:Couldn't resolve reference to Actor 'stock'. (line : 3 column : 46)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'this'. (line : 3 column : 70)
|
||||
ERROR:Couldn't resolve reference to ClassOrLabel 'product'. (line : 3 column : 75)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 74
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 1
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 75
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 89.5
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+145
@@ -0,0 +1,145 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:42:52
|
||||
* UniqueID: e6644809-f5a3-4705-8cdd-4206e476df36
|
||||
* Gherkinscenario name: Gherkinscenario 8: Disabling country_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: SystemA has country "United States"
|
||||
When: UserA wants to edit this country
|
||||
Then: UserA should be notified that it has been successfully edited
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: SystemA has country "United States"
|
||||
When: UserA wants to edit this country
|
||||
Then: UserA should be notified that it has been successfully edited
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
UserA should While assert that UserA $modals country has been edited$gal once Triggered $systemAU textured SystemAY United States$galribelowhinewherit
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
|
||||
ERROR:missing '.' at 'While' (line : 3 column : 14)
|
||||
ERROR:no viable alternative at input 'that' (line : 3 column : 20)
|
||||
ERROR:no viable alternative at input 'that' (line : 3 column : 27)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 32)
|
||||
ERROR:no viable alternative at input '$' (line : 3 column : 38)
|
||||
ERROR:no viable alternative at input 'modals' (line : 3 column : 39)
|
||||
ERROR:no viable alternative at input 'country' (line : 3 column : 46)
|
||||
ERROR:missing '.' at 'been' (line : 3 column : 58)
|
||||
ERROR:no viable alternative at input 'edited' (line : 3 column : 63)
|
||||
ERROR:missing '.' at 'gal' (line : 3 column : 70)
|
||||
ERROR:no viable alternative at input 'once' (line : 3 column : 74)
|
||||
ERROR:missing '.' at 'Triggered' (line : 3 column : 79)
|
||||
ERROR:no viable alternative at input '$' (line : 3 column : 89)
|
||||
ERROR:no viable alternative at input 'systemAU' (line : 3 column : 90)
|
||||
ERROR:no viable alternative at input 'textured' (line : 3 column : 99)
|
||||
ERROR:missing '.' at 'SystemAY' (line : 3 column : 108)
|
||||
ERROR:no viable alternative at input 'United' (line : 3 column : 117)
|
||||
ERROR:missing '.' at 'States' (line : 3 column : 124)
|
||||
ERROR:no viable alternative at input '$' (line : 3 column : 130)
|
||||
ERROR:no viable alternative at input 'galribelowhinewherit' (line : 3 column : 131)
|
||||
ERROR:Couldn't resolve reference to Actor 'assert'. (line : 3 column : 20)
|
||||
ERROR:Couldn't resolve reference to Actor 'modals'. (line : 3 column : 39)
|
||||
ERROR:Couldn't resolve reference to Actor 'been'. (line : 3 column : 58)
|
||||
ERROR:Couldn't resolve reference to Actor 'gal'. (line : 3 column : 70)
|
||||
ERROR:Couldn't resolve reference to Actor 'Triggered'. (line : 3 column : 79)
|
||||
ERROR:Couldn't resolve reference to Actor 'systemAU'. (line : 3 column : 90)
|
||||
ERROR:Couldn't resolve reference to Actor 'SystemAY'. (line : 3 column : 108)
|
||||
ERROR:Couldn't resolve reference to Actor 'States'. (line : 3 column : 124)
|
||||
ERROR:Couldn't resolve reference to Actor 'galribelowhinewherit'. (line : 3 column : 131)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 64
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,1
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 0
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 80
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 62.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+140
@@ -0,0 +1,140 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:41:57
|
||||
* UniqueID: cd538520-59c9-44be-9245-8363af6445cb
|
||||
* Gherkinscenario name: Gherkinscenario 8: Disabling country
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: SystemA has country "United States"
|
||||
When: UserA wants to edit this country
|
||||
Then: UserA should be notified that it has been successfully edited
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: SystemA has country "United States"
|
||||
When: UserA wants to edit this country
|
||||
Then: UserA should be notified that it has been successfully edited
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
While SystemA has country "United States" When UserA wants to edit this country UserA shall be notified that it has been successfully edited
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'to' (line : 3 column : 60)
|
||||
ERROR:missing ',' at 'edit' (line : 3 column : 63)
|
||||
ERROR:no viable alternative at input 'this' (line : 3 column : 68)
|
||||
ERROR:missing '.' at 'country' (line : 3 column : 73)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 81)
|
||||
ERROR:missing '.' at 'be' (line : 3 column : 93)
|
||||
ERROR:no viable alternative at input 'notified' (line : 3 column : 96)
|
||||
ERROR:missing '.' at 'it' (line : 3 column : 110)
|
||||
ERROR:no viable alternative at input 'has' (line : 3 column : 113)
|
||||
ERROR:no viable alternative at input 'been' (line : 3 column : 117)
|
||||
ERROR:no viable alternative at input 'successfully' (line : 3 column : 122)
|
||||
ERROR:missing '.' at 'edited' (line : 3 column : 135)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'country'. (line : 3 column : 19)
|
||||
ERROR:Couldn't resolve reference to Actor 'wants'. (line : 3 column : 54)
|
||||
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 48)
|
||||
ERROR:Couldn't resolve reference to Actor 'edit'. (line : 3 column : 63)
|
||||
ERROR:Couldn't resolve reference to Actor 'country'. (line : 3 column : 73)
|
||||
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 93)
|
||||
ERROR:Couldn't resolve reference to Actor 'it'. (line : 3 column : 110)
|
||||
ERROR:Couldn't resolve reference to Actor 'been'. (line : 3 column : 117)
|
||||
ERROR:Couldn't resolve reference to Actor 'edited'. (line : 3 column : 135)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 73
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 1
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 95
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 99.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+116
@@ -0,0 +1,116 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:45:13
|
||||
* UniqueID: a6c4e910-ed52-4352-a840-be0ead2b31f0
|
||||
* Gherkinscenario name: Gherkinscenario 9: Resending the verification email as a logged in user_Incorrect_
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA is logged in as "valkyrie@cain.com"
|
||||
When: UserA resends the verification email
|
||||
Then: UserA should be notified that the verification email has been sent
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: should or can
|
||||
$TRIGGER is for example a trigger action, nothing what a system does.
|
||||
|
||||
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
|
||||
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA is logged in as "valkyrie@cain.com"
|
||||
When: UserA resends the verification email
|
||||
Then: UserA should be notified that the verification email has been sent
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
=UserA should be notified that the verification email has been sent should UserA While UserA resends the verification email
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = -10
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:missing EOF at '=' (line : 3 column : 1)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 83
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 1
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 2
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 95
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 79.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+140
@@ -0,0 +1,140 @@
|
||||
|
||||
|
||||
# LLM Experiment Information
|
||||
* Prefix: Few-shot-learning
|
||||
* Datetime: 04_05_2024__14:43:43
|
||||
* UniqueID: 83374f34-15f4-4e37-a986-c0965cb51de1
|
||||
* Gherkinscenario name: Gherkinscenario 9: Resending the verification email as a logged in user
|
||||
* LLM-temperature: 1.3
|
||||
|
||||
|
||||
|
||||
## Gherkin Input
|
||||
```
|
||||
Given: UserA is logged in as "valkyrie@cain.com"
|
||||
When: UserA resends the verification email
|
||||
Then: UserA should be notified that the verification email has been sent
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Input prompt, technique: Few-Shot-Learning
|
||||
|
||||
|
||||
### Context
|
||||
We are translating Gherkin acceptance criteria into Rimay CNL.
|
||||
For this translation use the following information:
|
||||
|
||||
$ACTOR is for example a thing or a person.
|
||||
$MODAL_VERB is for the following word: must or shall.
|
||||
$TRIGGER is for example a trigger action, something what a system does.
|
||||
|
||||
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
|
||||
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
|
||||
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
|
||||
|
||||
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
|
||||
Replace the variables with the right text according the input acceptance criteria.
|
||||
In the following order:
|
||||
|
||||
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
|
||||
|
||||
The following example is a valid Rimay CNL:
|
||||
|
||||
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
|
||||
|
||||
With the usage of the following extra information:
|
||||
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
|
||||
|
||||
|
||||
|
||||
|
||||
### Input Data
|
||||
Use the following Gherkin acceptance criteria input:
|
||||
```
|
||||
Given: UserA is logged in as "valkyrie@cain.com"
|
||||
When: UserA resends the verification email
|
||||
Then: UserA should be notified that the verification email has been sent
|
||||
```
|
||||
|
||||
### Question:
|
||||
Can you translate the input above into the specified language Rimay CNL?
|
||||
|
||||
### Output indicator
|
||||
Output only the translated Rimay CNL text and nothing else!
|
||||
|
||||
|
||||
## Rimay Output
|
||||
```
|
||||
When UserA is logged in as "valkyrie@cain.com", UserA must resend the verification email. UserA should be notified that the verification email has been sent.
|
||||
```
|
||||
|
||||
|
||||
### Researcher score
|
||||
Researcher_score = 15
|
||||
|
||||
|
||||
|
||||
|
||||
## DSL-Rimay Verification
|
||||
```
|
||||
ERROR:no viable alternative at input 'is' (line : 3 column : 12)
|
||||
ERROR:missing ',' at 'logged' (line : 3 column : 15)
|
||||
ERROR:no viable alternative at input 'in' (line : 3 column : 22)
|
||||
ERROR:no viable alternative at input 'UserA' (line : 3 column : 49)
|
||||
ERROR:no viable alternative at input 'resend' (line : 3 column : 60)
|
||||
ERROR:no viable alternative at input 'the' (line : 3 column : 67)
|
||||
ERROR:missing '.' at 'verification' (line : 3 column : 71)
|
||||
ERROR:no viable alternative at input 'email' (line : 3 column : 84)
|
||||
ERROR:no viable alternative at input 'should' (line : 3 column : 97)
|
||||
ERROR:missing '.' at 'be' (line : 3 column : 104)
|
||||
ERROR:no viable alternative at input 'notified' (line : 3 column : 107)
|
||||
ERROR:missing '.' at 'the' (line : 3 column : 121)
|
||||
ERROR:no viable alternative at input 'email' (line : 3 column : 138)
|
||||
ERROR:missing '.' at 'been' (line : 3 column : 148)
|
||||
ERROR:no viable alternative at input 'sent' (line : 3 column : 153)
|
||||
ERROR:Couldn't resolve reference to Actor 'logged'. (line : 3 column : 15)
|
||||
ERROR:Couldn't resolve reference to Actor 'resend'. (line : 3 column : 60)
|
||||
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 71)
|
||||
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 104)
|
||||
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 125)
|
||||
ERROR:Couldn't resolve reference to Actor 'been'. (line : 3 column : 148)
|
||||
|
||||
```
|
||||
### DSL-Rimay Score
|
||||
DSL_Score: 73
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
|
||||
## Rimay Paska Verification
|
||||
```
|
||||
File generated_rimay.csv.csv
|
||||
Smells found in this document:
|
||||
Non-atomic requirement: 0
|
||||
Incomplete requirement: 0
|
||||
Incorrect order requirement: 0
|
||||
Coordination ambiguity: 0
|
||||
Not requirement: ,0
|
||||
Incomplete condition: 0
|
||||
Incomplete system response: 0
|
||||
Passive voice: 1
|
||||
Not precise verb: 0
|
||||
Number Req with smells: 1
|
||||
|
||||
```
|
||||
### Paska Score
|
||||
PASKA_Score: 95
|
||||
|
||||
Progressbar: 
|
||||
|
||||
|
||||
|
||||
### Final Score
|
||||
Final_score = 99.0
|
||||
|
||||
Progressbar: 
|
||||
|
||||
+2
-2
@@ -32,7 +32,7 @@ def convert_single_gherkin_to_rimay(scenario_name: str, input_text:str, translat
|
||||
```
|
||||
"""
|
||||
current_logger.append_result(pre_content)
|
||||
response = ask_different_prompts(input_text.strip(), translation_type, current_logger, incorrect, temperature)
|
||||
response = ask_different_prompts(input_text.strip(), translation_type, current_logger, temperature, incorrect)
|
||||
researcher_score(input_text, response, current_logger)
|
||||
rimay_check(response, current_logger)
|
||||
|
||||
@@ -141,7 +141,7 @@ def results(path):
|
||||
|
||||
|
||||
],
|
||||
}).classes('w-full h-200')
|
||||
}).classes('w-full h-400')
|
||||
|
||||
def update():
|
||||
chart.options['series'][0]['data'][0] = random()
|
||||
|
||||
Reference in New Issue
Block a user