Added some extra readme and new folder structures

This commit is contained in:
koenieeeee
2024-03-19 23:07:30 +01:00
parent 5f034d58fd
commit eac3fb524e
19 changed files with 1247 additions and 3 deletions
+7 -2
View File
@@ -63,6 +63,11 @@ sudo apt-get update
sudo apt-get install bellsoft-java8-runtime-full
```
Also download the following distsim tagger file:
```
https://github.com/amiangshu/SentiSE/blob/master/edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger
```
And see the following file for needed modifications to paths: `/LLPTE/rimay_verification.py`
Also, there is no error when the directory of preproccesing tooling is not found, be aware. And use full paths for everything.
Also, there is no error when the directory of preproccesing tooling is not found, be aware. And use full paths for everything.
I precompiled the jar files for Paska and DSL Rimay.
+1 -1
View File
@@ -63,7 +63,7 @@ class GherkinData(ProcessTextFile, LLM_prompt_data):
self.gherkin_list = self.load()
def load(self):
return super()._load_xml_file("input_dataset.xml")
return super()._load_xml_file("input_dataset/input_dataset.xml")
#save not implemented
+7
View File
@@ -0,0 +1,7 @@
# Dataset Gherkin
The dataset in `input_dataset.xml` is custom made and derrived from the `Open_Source_Projects_GitHub_US_AC_Analysis.xlsx` file
There is an original tag and an simplified tag in the dataset xml file
The simplified tag is used to make it easer for the LLM to translate Gherkin into Rimay. I made some adjustments to the sentence.
View File
@@ -0,0 +1,142 @@
# LLM Experiment Information
* Prefix: Chain-of-thought
* Datetime: 03_19_2024__22:53:14
* UniqueID: 5ee2bdf3-bf47-45be-a60a-27490ca410f1
## Gherkin Input
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
## Input prompt, technique: Chain of Thought
Learn from the following example, the output information for the CNL Rimay is:
This part is the start of the example you have to learn from:
The input information is as follows:
Acceptance Criteria 1 in Gherkin, input:
Gherkin:
Scenario: Create an Order
Given an Order of type Subscription_Order does not exist in OI of type Order_Issuer
When OI Create Order,
Then Order exists in OI
And the property settlement_method of Order is equal to FOP
Rimay output 1:
When the Order_Issuer (OI ) executes Create a Subscription_Order, if the Order does not exist, then the Order is created and its settlement_method is set to FOP.
Rimay uses the following structure:
When the? Actor <Action> (every "Frequency")? ,|then the? Actor must <Action> (every "Text")?.
Ending for example
Translate the following input acceptance criteria Gherkin, according to the example above, to Rimay:
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
Given a verification email has already been sent to "valkyrie@cain.com", when the user tries to verify their account using the link from this email, then the user should be notified that the verification was successful.
```
## Rimay DSL Verification
```
ERROR:no viable alternative at input 'a' (line : 3 column : 7)
ERROR:missing '.' at 'verification' (line : 3 column : 9)
ERROR:no viable alternative at input 'email' (line : 3 column : 22)
ERROR:missing '.' at 'already' (line : 3 column : 32)
ERROR:no viable alternative at input 'been' (line : 3 column : 40)
ERROR:missing '.' at 'sent' (line : 3 column : 45)
ERROR:no viable alternative at input 'to' (line : 3 column : 50)
ERROR:no viable alternative at input 'when' (line : 3 column : 74)
ERROR:no viable alternative at input 'tries' (line : 3 column : 88)
ERROR:no viable alternative at input 'to' (line : 3 column : 94)
ERROR:no viable alternative at input 'verify' (line : 3 column : 97)
ERROR:no viable alternative at input 'their' (line : 3 column : 104)
ERROR:missing '.' at 'account' (line : 3 column : 110)
ERROR:no viable alternative at input 'using' (line : 3 column : 118)
ERROR:no viable alternative at input 'the' (line : 3 column : 124)
ERROR:mismatched input 'link' expecting RULE_ID (line : 3 column : 128)
ERROR:missing '.' at 'this' (line : 3 column : 138)
ERROR:no viable alternative at input 'email' (line : 3 column : 143)
ERROR:missing '.' at 'the' (line : 3 column : 155)
ERROR:no viable alternative at input 'should' (line : 3 column : 164)
ERROR:missing '.' at 'be' (line : 3 column : 171)
ERROR:no viable alternative at input 'notified' (line : 3 column : 174)
ERROR:missing '.' at 'the' (line : 3 column : 188)
ERROR:no viable alternative at input 'was' (line : 3 column : 205)
ERROR:extraneous input 'successful' expecting '.' (line : 3 column : 209)
ERROR:Couldn't resolve reference to Actor 'Given'. (line : 3 column : 1)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 9)
ERROR:Couldn't resolve reference to Actor 'already'. (line : 3 column : 32)
ERROR:Couldn't resolve reference to Actor 'sent'. (line : 3 column : 45)
ERROR:Couldn't resolve reference to Actor 'user'. (line : 3 column : 83)
ERROR:Couldn't resolve reference to Actor 'tries'. (line : 3 column : 88)
ERROR:Couldn't resolve reference to Actor 'verify'. (line : 3 column : 97)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 110)
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 138)
ERROR:Couldn't resolve reference to Actor 'user'. (line : 3 column : 159)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 171)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 192)
```
### Score
Score: 57
Progressbar: ![57%](https://progress-bar.dev/57)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 1
Not precise verb: 0
Number Req with smells: 1
```
### Score
Score: 96
Progressbar: ![96%](https://progress-bar.dev/96)
### Final Score
Final_score = 76.5
@@ -0,0 +1,142 @@
# LLM Experiment Information
* Prefix: Chain-of-thought
* Datetime: 03_19_2024__22:57:31
* UniqueID: d6283b96-ded7-4bbd-a9c9-d6f25fc23b06
## Gherkin Input
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
## Input prompt, technique: Chain of Thought
Learn from the following example, the output information for the CNL Rimay is:
This part is the start of the example you have to learn from:
The input information is as follows:
Acceptance Criteria 1 in Gherkin, input:
Gherkin:
Scenario: Create an Order
Given an Order of type Subscription_Order does not exist in OI of type Order_Issuer
When OI Create Order,
Then Order exists in OI
And the property settlement_method of Order is equal to FOP
Rimay output 1:
When the Order_Issuer (OI ) executes Create a Subscription_Order, if the Order does not exist, then the Order is created and its settlement_method is set to FOP.
Rimay uses the following structure:
When the? Actor <Action> (every "Frequency")? ,|then the? Actor must <Action> (every "Text")?.
Ending for example
Translate the following input acceptance criteria Gherkin, according to the example above, to Rimay:
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
Given a verification email has already been sent to "valkyrie@cain.com", when the user tries to verify their account using the link from this email, then the user should be notified that the verification was successful.
```
## Rimay DSL Verification
```
ERROR:no viable alternative at input 'a' (line : 3 column : 7)
ERROR:missing '.' at 'verification' (line : 3 column : 9)
ERROR:no viable alternative at input 'email' (line : 3 column : 22)
ERROR:missing '.' at 'already' (line : 3 column : 32)
ERROR:no viable alternative at input 'been' (line : 3 column : 40)
ERROR:missing '.' at 'sent' (line : 3 column : 45)
ERROR:no viable alternative at input 'to' (line : 3 column : 50)
ERROR:no viable alternative at input 'when' (line : 3 column : 74)
ERROR:no viable alternative at input 'tries' (line : 3 column : 88)
ERROR:no viable alternative at input 'to' (line : 3 column : 94)
ERROR:no viable alternative at input 'verify' (line : 3 column : 97)
ERROR:no viable alternative at input 'their' (line : 3 column : 104)
ERROR:missing '.' at 'account' (line : 3 column : 110)
ERROR:no viable alternative at input 'using' (line : 3 column : 118)
ERROR:no viable alternative at input 'the' (line : 3 column : 124)
ERROR:mismatched input 'link' expecting RULE_ID (line : 3 column : 128)
ERROR:missing '.' at 'this' (line : 3 column : 138)
ERROR:no viable alternative at input 'email' (line : 3 column : 143)
ERROR:missing '.' at 'the' (line : 3 column : 155)
ERROR:no viable alternative at input 'should' (line : 3 column : 164)
ERROR:missing '.' at 'be' (line : 3 column : 171)
ERROR:no viable alternative at input 'notified' (line : 3 column : 174)
ERROR:missing '.' at 'the' (line : 3 column : 188)
ERROR:no viable alternative at input 'was' (line : 3 column : 205)
ERROR:extraneous input 'successful' expecting '.' (line : 3 column : 209)
ERROR:Couldn't resolve reference to Actor 'Given'. (line : 3 column : 1)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 9)
ERROR:Couldn't resolve reference to Actor 'already'. (line : 3 column : 32)
ERROR:Couldn't resolve reference to Actor 'sent'. (line : 3 column : 45)
ERROR:Couldn't resolve reference to Actor 'user'. (line : 3 column : 83)
ERROR:Couldn't resolve reference to Actor 'tries'. (line : 3 column : 88)
ERROR:Couldn't resolve reference to Actor 'verify'. (line : 3 column : 97)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 110)
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 138)
ERROR:Couldn't resolve reference to Actor 'user'. (line : 3 column : 159)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 171)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 192)
```
### Score
Score: 57
Progressbar: ![57%](https://progress-bar.dev/57)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 1
Not precise verb: 0
Number Req with smells: 1
```
### Score
Score: 96
Progressbar: ![96%](https://progress-bar.dev/96)
### Final Score
Final_score = 76.5
@@ -0,0 +1,125 @@
# LLM Experiment Information
* Prefix: Chain-of-thought
* Datetime: 03_19_2024__22:55:23
* UniqueID: 3b9d1772-f5d3-4d15-8d53-a2b47e43f0ef
## Gherkin Input
```
Given: I am editing the address of "Fletcher Ren"
When: I remove the street
Then: I should still be on the "Fletcher Ren" address edit page
```
## Input prompt, technique: Chain of Thought
Learn from the following example, the output information for the CNL Rimay is:
This part is the start of the example you have to learn from:
The input information is as follows:
Acceptance Criteria 1 in Gherkin, input:
Gherkin:
Scenario: Create an Order
Given an Order of type Subscription_Order does not exist in OI of type Order_Issuer
When OI Create Order,
Then Order exists in OI
And the property settlement_method of Order is equal to FOP
Rimay output 1:
When the Order_Issuer (OI ) executes Create a Subscription_Order, if the Order does not exist, then the Order is created and its settlement_method is set to FOP.
Rimay uses the following structure:
When the? Actor <Action> (every "Frequency")? ,|then the? Actor must <Action> (every "Text")?.
Ending for example
Translate the following input acceptance criteria Gherkin, according to the example above, to Rimay:
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: I am editing the address of "Fletcher Ren"
When: I remove the street
Then: I should still be on the "Fletcher Ren" address edit page
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
Given: When editing the address of "Fletcher Ren", if the street is removed, then I should remain on the "Fletcher Ren" address edit page.
```
## Rimay DSL Verification
```
ERROR:no viable alternative at input ':' (line : 3 column : 6)
ERROR:no viable alternative at input 'When' (line : 3 column : 8)
ERROR:no viable alternative at input 'the' (line : 3 column : 21)
ERROR:no viable alternative at input 'of' (line : 3 column : 33)
ERROR:no viable alternative at input 'if' (line : 3 column : 52)
ERROR:no viable alternative at input ',' (line : 3 column : 76)
ERROR:no viable alternative at input 'should' (line : 3 column : 85)
ERROR:missing '.' at 'remain' (line : 3 column : 92)
ERROR:no viable alternative at input 'on' (line : 3 column : 99)
ERROR:no viable alternative at input 'the' (line : 3 column : 102)
ERROR:extraneous input '"Fletcher Ren"' expecting RULE_ID (line : 3 column : 106)
ERROR:no viable alternative at input 'edit' (line : 3 column : 129)
ERROR:Couldn't resolve reference to Actor 'Given'. (line : 3 column : 1)
ERROR:Couldn't resolve reference to Actor 'editing'. (line : 3 column : 13)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 25)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'street'. (line : 3 column : 59)
ERROR:Couldn't resolve reference to Quantifier 'removed'. (line : 3 column : 69)
ERROR:Couldn't resolve reference to Actor 'I'. (line : 3 column : 83)
ERROR:Couldn't resolve reference to Actor 'remain'. (line : 3 column : 92)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 121)
```
### Score
Score: 74
Progressbar: ![74%](https://progress-bar.dev/74)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 2
Incomplete system response: 0
Passive voice: 1
Not precise verb: 0
Number Req with smells: 1
```
### Score
Score: 92
Progressbar: ![92%](https://progress-bar.dev/92)
### Final Score
Final_score = 83.0
@@ -0,0 +1,166 @@
# LLM Experiment Information
* Prefix: Few-shot learning
* Datetime: 03_19_2024__22:56:48
* UniqueID: 09643107-0bc8-43c8-9a9c-2d60d5d3df0e
## Gherkin Input
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
## Input prompt, technique: Few-Shot Learning
### Context
The following syntax structure is called Rimay, this is a CNL (Controlled Natural Language):
$ACTOR is for example a thing or a person.
$MODAL_VERB is for example the following word: must and shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHEN_STRUCTURE is the word 'When' followed by the variable defined $TRIGGER.
$SYSTEM_RESPONSE is something that the syst
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
really important rule, make only use of the following word list:
This list of variables is important and can be used to formulete Rimay:
Only make use of the following words:
is, are, equal, equals, greater than, less than, has, have, contain, contains, do, does, every,
before, after, if, and, or, not, of, permit, permits, accept, acepts, allow, allows, is, are, shall, must,
a, an, the, do, the, following, actions, exclude, excludes, instruct, instructs,
allow, allows, authorize, authorizes, request, requests,
start, starts, begin, begins, cancel, cancels, hide, hides,
restore, restores, compute, computes, publish, publishes, enable, disable,
enforce, enforces, create, creates, generate, generates, replace, replaces,
prevent, prevents, download, downloads, aggregate, aggregates,
interrupt, interrupts, include, includes, store, stores, property, properties, value, values,
limit, limits, restrict, restricts, reduce, reduces, migrate, migrates,
add, adds, link, links, neglect, neglects, ignore, ignores, accept, accepts, receive, receives, retrieve, retrieves,
close, closes, reverse, reverses, insert, inserts, display, displays, show, shows,
extract, extracts, remove, removes, delete, deletes, deduct, deducts, report, reports, propose, proposes,
detect, detects, select, selects, unselect, unselects, return, returns, send, sends, forward, forward, pass, passes, export, exports,
concatenate, concatenates, synchronize, synchronizes, discard, discards, copy, copies,
convert, converts, change, changes, transform, transforms, update, updates, set, sets,
use, uses, apply, applies, validate, validates, check, checks, calculate, calculates, recalculate, recalculates
establish, establishes, search, searches, split, splits, stop, stops, finish, finishes
subscribe, subscribes, upload, uploads
Important extra rule: if the word is not defined in the list above, put if between quotes.
DO NOT WRITE ANY OTHER WORDS THAN THE DEFINED WORD LIST!
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
Classes: class Instruction := description record, class Y
THIS IS NOT VALID RIMAY:
When UserA navigates to the login page, UserA clicks on button, SystemA must refresh the page.
The following example would be valid:
When UserA "navigates to the login page", UserA "clicks on button", SystemA must "refresh the page".
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
When User "I" tries to verify their account using the link from the verification email sent to "valkyrie@cain.com", User "I" must be notified that the verification was successful.
```
## Rimay DSL Verification
```
ERROR:no viable alternative at input 'tries' (line : 3 column : 15)
ERROR:no viable alternative at input 'to' (line : 3 column : 21)
ERROR:no viable alternative at input 'verify' (line : 3 column : 24)
ERROR:no viable alternative at input 'their' (line : 3 column : 31)
ERROR:missing '.' at 'account' (line : 3 column : 37)
ERROR:no viable alternative at input 'using' (line : 3 column : 45)
ERROR:no viable alternative at input 'the' (line : 3 column : 51)
ERROR:mismatched input 'link' expecting RULE_ID (line : 3 column : 55)
ERROR:missing '.' at 'the' (line : 3 column : 65)
ERROR:no viable alternative at input 'email' (line : 3 column : 82)
ERROR:missing '.' at 'sent' (line : 3 column : 88)
ERROR:no viable alternative at input 'to' (line : 3 column : 93)
ERROR:no viable alternative at input 'User' (line : 3 column : 117)
ERROR:no viable alternative at input '"I"' (line : 3 column : 122)
ERROR:no viable alternative at input 'be' (line : 3 column : 131)
ERROR:no viable alternative at input 'notified' (line : 3 column : 134)
ERROR:missing '.' at 'the' (line : 3 column : 148)
ERROR:no viable alternative at input 'was' (line : 3 column : 165)
ERROR:extraneous input 'successful' expecting '.' (line : 3 column : 169)
ERROR:Couldn't resolve reference to Actor 'User'. (line : 3 column : 6)
ERROR:Couldn't resolve reference to Actor 'tries'. (line : 3 column : 15)
ERROR:Couldn't resolve reference to Actor 'verify'. (line : 3 column : 24)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 37)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 69)
ERROR:Couldn't resolve reference to Actor 'sent'. (line : 3 column : 88)
ERROR:Couldn't resolve reference to Actor 'User'. (line : 3 column : 117)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 131)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 152)
```
### Score
Score: 66
Progressbar: ![66%](https://progress-bar.dev/66)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 1
Not precise verb: 0
Number Req with smells: 1
```
### Score
Score: 96
Progressbar: ![96%](https://progress-bar.dev/96)
### Final Score
Final_score = 81.0
@@ -0,0 +1,162 @@
# LLM Experiment Information
* Prefix: Few-shot learning
* Datetime: 03_19_2024__22:54:40
* UniqueID: 083e8c75-643c-4688-a16a-0dd018d10e41
## Gherkin Input
```
Given: I am editing the address of "Fletcher Ren"
When: I remove the street
Then: I should still be on the "Fletcher Ren" address edit page
```
## Input prompt, technique: Few-Shot Learning
### Context
The following syntax structure is called Rimay, this is a CNL (Controlled Natural Language):
$ACTOR is for example a thing or a person.
$MODAL_VERB is for example the following word: must and shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHEN_STRUCTURE is the word 'When' followed by the variable defined $TRIGGER.
$SYSTEM_RESPONSE is something that the syst
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
really important rule, make only use of the following word list:
This list of variables is important and can be used to formulete Rimay:
Only make use of the following words:
is, are, equal, equals, greater than, less than, has, have, contain, contains, do, does, every,
before, after, if, and, or, not, of, permit, permits, accept, acepts, allow, allows, is, are, shall, must,
a, an, the, do, the, following, actions, exclude, excludes, instruct, instructs,
allow, allows, authorize, authorizes, request, requests,
start, starts, begin, begins, cancel, cancels, hide, hides,
restore, restores, compute, computes, publish, publishes, enable, disable,
enforce, enforces, create, creates, generate, generates, replace, replaces,
prevent, prevents, download, downloads, aggregate, aggregates,
interrupt, interrupts, include, includes, store, stores, property, properties, value, values,
limit, limits, restrict, restricts, reduce, reduces, migrate, migrates,
add, adds, link, links, neglect, neglects, ignore, ignores, accept, accepts, receive, receives, retrieve, retrieves,
close, closes, reverse, reverses, insert, inserts, display, displays, show, shows,
extract, extracts, remove, removes, delete, deletes, deduct, deducts, report, reports, propose, proposes,
detect, detects, select, selects, unselect, unselects, return, returns, send, sends, forward, forward, pass, passes, export, exports,
concatenate, concatenates, synchronize, synchronizes, discard, discards, copy, copies,
convert, converts, change, changes, transform, transforms, update, updates, set, sets,
use, uses, apply, applies, validate, validates, check, checks, calculate, calculates, recalculate, recalculates
establish, establishes, search, searches, split, splits, stop, stops, finish, finishes
subscribe, subscribes, upload, uploads
Important extra rule: if the word is not defined in the list above, put if between quotes.
DO NOT WRITE ANY OTHER WORDS THAN THE DEFINED WORD LIST!
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
Classes: class Instruction := description record, class Y
THIS IS NOT VALID RIMAY:
When UserA navigates to the login page, UserA clicks on button, SystemA must refresh the page.
The following example would be valid:
When UserA "navigates to the login page", UserA "clicks on button", SystemA must "refresh the page".
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: I am editing the address of "Fletcher Ren"
When: I remove the street
Then: I should still be on the "Fletcher Ren" address edit page
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
When User "I" edits the address of "Fletcher Ren", User "I" removes the street, User "I" should still be on the "Fletcher Ren" address edit page.
```
## Rimay DSL Verification
```
ERROR:no viable alternative at input 'edits' (line : 3 column : 15)
ERROR:no viable alternative at input 'the' (line : 3 column : 21)
ERROR:missing '.' at 'address' (line : 3 column : 25)
ERROR:no viable alternative at input 'of' (line : 3 column : 33)
ERROR:no viable alternative at input 'User' (line : 3 column : 52)
ERROR:no viable alternative at input '"I"' (line : 3 column : 57)
ERROR:mismatched input 'User' expecting 'or' (line : 3 column : 81)
ERROR:no viable alternative at input '"I"' (line : 3 column : 86)
ERROR:no viable alternative at input 'should' (line : 3 column : 90)
ERROR:no viable alternative at input 'still' (line : 3 column : 97)
ERROR:missing '.' at 'be' (line : 3 column : 103)
ERROR:no viable alternative at input 'on' (line : 3 column : 106)
ERROR:no viable alternative at input 'the' (line : 3 column : 109)
ERROR:extraneous input '"Fletcher Ren"' expecting RULE_ID (line : 3 column : 113)
ERROR:no viable alternative at input 'edit' (line : 3 column : 136)
ERROR:Couldn't resolve reference to Actor 'User'. (line : 3 column : 6)
ERROR:Couldn't resolve reference to Actor 'edits'. (line : 3 column : 15)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 25)
ERROR:Couldn't resolve reference to Actor 'User'. (line : 3 column : 52)
ERROR:Couldn't resolve reference to PropertyOrLabel 'street'. (line : 3 column : 73)
ERROR:Couldn't resolve reference to Actor 'User'. (line : 3 column : 81)
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 90)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 103)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 128)
```
### Score
Score: 70
Progressbar: ![70%](https://progress-bar.dev/70)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Score
Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 85.0
@@ -0,0 +1,161 @@
# LLM Experiment Information
* Prefix: Few-shot learning
* Datetime: 03_19_2024__22:52:17
* UniqueID: dc906f92-4048-431b-9723-5ac5ee37c5a3
## Gherkin Input
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
## Input prompt, technique: Few-Shot Learning
### Context
The following syntax structure is called Rimay, this is a CNL (Controlled Natural Language):
$ACTOR is for example a thing or a person.
$MODAL_VERB is for example the following word: must and shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHEN_STRUCTURE is the word 'When' followed by the variable defined $TRIGGER.
$SYSTEM_RESPONSE is something that the syst
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
really important rule, make only use of the following word list:
This list of variables is important and can be used to formulete Rimay:
Only make use of the following words:
is, are, equal, equals, greater than, less than, has, have, contain, contains, do, does, every,
before, after, if, and, or, not, of, permit, permits, accept, acepts, allow, allows, is, are, shall, must,
a, an, the, do, the, following, actions, exclude, excludes, instruct, instructs,
allow, allows, authorize, authorizes, request, requests,
start, starts, begin, begins, cancel, cancels, hide, hides,
restore, restores, compute, computes, publish, publishes, enable, disable,
enforce, enforces, create, creates, generate, generates, replace, replaces,
prevent, prevents, download, downloads, aggregate, aggregates,
interrupt, interrupts, include, includes, store, stores, property, properties, value, values,
limit, limits, restrict, restricts, reduce, reduces, migrate, migrates,
add, adds, link, links, neglect, neglects, ignore, ignores, accept, accepts, receive, receives, retrieve, retrieves,
close, closes, reverse, reverses, insert, inserts, display, displays, show, shows,
extract, extracts, remove, removes, delete, deletes, deduct, deducts, report, reports, propose, proposes,
detect, detects, select, selects, unselect, unselects, return, returns, send, sends, forward, forward, pass, passes, export, exports,
concatenate, concatenates, synchronize, synchronizes, discard, discards, copy, copies,
convert, converts, change, changes, transform, transforms, update, updates, set, sets,
use, uses, apply, applies, validate, validates, check, checks, calculate, calculates, recalculate, recalculates
establish, establishes, search, searches, split, splits, stop, stops, finish, finishes
subscribe, subscribes, upload, uploads
Important extra rule: if the word is not defined in the list above, put if between quotes.
DO NOT WRITE ANY OTHER WORDS THAN THE DEFINED WORD LIST!
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
Classes: class Instruction := description record, class Y
THIS IS NOT VALID RIMAY:
When UserA navigates to the login page, UserA clicks on button, SystemA must refresh the page.
The following example would be valid:
When UserA "navigates to the login page", UserA "clicks on button", SystemA must "refresh the page".
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
When User "valkyrie@cain.com" receives a "verification email", User must try to "verify account" using the link from this email. System should notify User that the verification was successful.
```
## Rimay DSL Verification
```
ERROR:no viable alternative at input 'receives' (line : 3 column : 31)
ERROR:missing ',' at 'a' (line : 3 column : 40)
ERROR:mismatched input '"verification email"' expecting RULE_ID (line : 3 column : 42)
ERROR:missing '.' at 'User' (line : 3 column : 64)
ERROR:no viable alternative at input 'try' (line : 3 column : 74)
ERROR:no viable alternative at input 'to' (line : 3 column : 78)
ERROR:no viable alternative at input 'the' (line : 3 column : 104)
ERROR:mismatched input 'link' expecting RULE_ID (line : 3 column : 108)
ERROR:missing '.' at 'this' (line : 3 column : 118)
ERROR:no viable alternative at input 'email' (line : 3 column : 123)
ERROR:no viable alternative at input 'should' (line : 3 column : 137)
ERROR:missing '.' at 'User' (line : 3 column : 151)
ERROR:no viable alternative at input 'that' (line : 3 column : 156)
ERROR:no viable alternative at input 'the' (line : 3 column : 161)
ERROR:no viable alternative at input 'was' (line : 3 column : 178)
ERROR:extraneous input 'successful' expecting '.' (line : 3 column : 182)
ERROR:Couldn't resolve reference to Actor 'User'. (line : 3 column : 6)
ERROR:Couldn't resolve reference to Actor 'User'. (line : 3 column : 64)
ERROR:Couldn't resolve reference to Actor 'try'. (line : 3 column : 74)
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 118)
ERROR:Couldn't resolve reference to Actor 'System'. (line : 3 column : 130)
ERROR:Couldn't resolve reference to Actor 'User'. (line : 3 column : 151)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 165)
```
### Score
Score: 71
Progressbar: ![71%](https://progress-bar.dev/71)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Score
Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 85.5
+10
View File
@@ -0,0 +1,10 @@
# Log directory
This directory contains all the logging with input data, output data and prompt send to the Large-Language-Model ChatGPT.
Also the Paska output and DSL Rimay output is logged.
The online progressbar api is used to render a visible progressbar.
## Todo add score explanation.
@@ -0,0 +1,107 @@
# LLM Experiment Information
* Prefix: Role play
* Datetime: 03_19_2024__22:56:06
* UniqueID: 6e325bd9-501c-4d15-bfde-c73c8e3ecd91
## Gherkin Input
```
Given: I am editing the address of "Fletcher Ren"
When: I remove the street
Then: I should still be on the "Fletcher Ren" address edit page
```
## Input prompt, technique: Role Play
Behave like a person who writes the Controlled Natural Language (CNL) Rimay.
Rimay is a language that is defined in different components.
For now we only highlight the following components:
Mandatory:
$ACTOR = SystemA
$MODAL_VERB = should, can, could, might, will
$SYSTEM_RESPONSE = SYSTEM_RESPONSE_EXPRESSION | RESPONSE_BLOCK_ITEMIZED
$SYSTEM_RESPONSE_EXPRESSION: ATOMIC_SYSTEM_RESPONSE ((,|, and|, or|and|or)? ATOMIC_SYSTEM_RESPONSE)* ATOMIC_SYSTEM_RESPONSE: ACTION_PHRASE (every TEXT )? RESPONSE_BLOCK_ITEMIZED: do the following actions (in sequence)? : BULLET ATOMIC_SYSTEM_RESPONSE ((,|, and|, or|and|or)? BULLET ATOMIC_SYSTEM_RESPONSE)*
Optional:
$WHEN_STRUCTURE : When TRIGGER
Use for ACTOR for example SystemA and use for MODAL_VERB for example the following words: should, can, could, might, will
$ACTOR, $MODAL_VERB and $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Rimay CNL: $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
Now I want you to answer as the person who writes Rimay to translate this acceptance criteria into Rimay
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: I am editing the address of "Fletcher Ren"
When: I remove the street
Then: I should still be on the "Fletcher Ren" address edit page
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
$ACTOR SystemA $MODAL_VERB should $SYSTEM_RESPONSE do the following actions in sequence:
- I am editing the address of "Fletcher Ren"
- I remove the street
- I should still be on the "Fletcher Ren" address edit page
```
## Rimay DSL Verification
```
ERROR:missing EOF at '$' (line : 3 column : 1)
```
### Score
Score: 63
Progressbar: ![63%](https://progress-bar.dev/63)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 1
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 1
Number Req with smells: 1
```
### Score
Score: 94
Progressbar: ![94%](https://progress-bar.dev/94)
### Final Score
Final_score = 78.5
@@ -0,0 +1,110 @@
# LLM Experiment Information
* Prefix: Role play
* Datetime: 03_19_2024__22:53:57
* UniqueID: 71f168d7-5242-4c4c-a828-6cd3b74b3d0b
## Gherkin Input
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
## Input prompt, technique: Role Play
Behave like a person who writes the Controlled Natural Language (CNL) Rimay.
Rimay is a language that is defined in different components.
For now we only highlight the following components:
Mandatory:
$ACTOR = SystemA
$MODAL_VERB = should, can, could, might, will
$SYSTEM_RESPONSE = SYSTEM_RESPONSE_EXPRESSION | RESPONSE_BLOCK_ITEMIZED
$SYSTEM_RESPONSE_EXPRESSION: ATOMIC_SYSTEM_RESPONSE ((,|, and|, or|and|or)? ATOMIC_SYSTEM_RESPONSE)* ATOMIC_SYSTEM_RESPONSE: ACTION_PHRASE (every TEXT )? RESPONSE_BLOCK_ITEMIZED: do the following actions (in sequence)? : BULLET ATOMIC_SYSTEM_RESPONSE ((,|, and|, or|and|or)? BULLET ATOMIC_SYSTEM_RESPONSE)*
Optional:
$WHEN_STRUCTURE : When TRIGGER
Use for ACTOR for example SystemA and use for MODAL_VERB for example the following words: should, can, could, might, will
$ACTOR, $MODAL_VERB and $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Rimay CNL: $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
Now I want you to answer as the person who writes Rimay to translate this acceptance criteria into Rimay
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
$ACTOR = SystemA
$MODAL_VERB = should
$ACTOR should $SYSTEM_RESPONSE do the following actions in sequence:
- Verify that a verification email has already been sent to "valkyrie@cain.com"
- Try to verify the account using the link from this email
- Notify that the verification was successful
```
## Rimay DSL Verification
```
ERROR:missing EOF at '$' (line : 3 column : 1)
```
### Score
Score: 63
Progressbar: ![63%](https://progress-bar.dev/63)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 1
Number Req with smells: 1
```
### Score
Score: 96
Progressbar: ![96%](https://progress-bar.dev/96)
### Final Score
Final_score = 79.5
@@ -0,0 +1,107 @@
# LLM Experiment Information
* Prefix: Role play
* Datetime: 03_19_2024__22:58:14
* UniqueID: 400b7b21-d4bb-4d07-9f8e-de4468a6af15
## Gherkin Input
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
## Input prompt, technique: Role Play
Behave like a person who writes the Controlled Natural Language (CNL) Rimay.
Rimay is a language that is defined in different components.
For now we only highlight the following components:
Mandatory:
$ACTOR = SystemA
$MODAL_VERB = should, can, could, might, will
$SYSTEM_RESPONSE = SYSTEM_RESPONSE_EXPRESSION | RESPONSE_BLOCK_ITEMIZED
$SYSTEM_RESPONSE_EXPRESSION: ATOMIC_SYSTEM_RESPONSE ((,|, and|, or|and|or)? ATOMIC_SYSTEM_RESPONSE)* ATOMIC_SYSTEM_RESPONSE: ACTION_PHRASE (every TEXT )? RESPONSE_BLOCK_ITEMIZED: do the following actions (in sequence)? : BULLET ATOMIC_SYSTEM_RESPONSE ((,|, and|, or|and|or)? BULLET ATOMIC_SYSTEM_RESPONSE)*
Optional:
$WHEN_STRUCTURE : When TRIGGER
Use for ACTOR for example SystemA and use for MODAL_VERB for example the following words: should, can, could, might, will
$ACTOR, $MODAL_VERB and $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Rimay CNL: $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
Now I want you to answer as the person who writes Rimay to translate this acceptance criteria into Rimay
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: a verification email has already been sent to "valkyrie@cain.com"
When: I try to verify my account using the link from this email
Then: I should be notified that the verification was successful
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only your translated Rimay CNL text and nothing else!
## Rimay Output
```
$ACTOR SystemA $MODAL_VERB should $SYSTEM_RESPONSE do the following actions in sequence:
- Verify that a verification email has already been sent to "valkyrie@cain.com"
- Try to verify the account using the link from this email
- Notify that the verification was successful
```
## Rimay DSL Verification
```
ERROR:missing EOF at '$' (line : 3 column : 1)
```
### Score
Score: 63
Progressbar: ![63%](https://progress-bar.dev/63)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 1
Number Req with smells: 1
```
### Score
Score: 96
Progressbar: ![96%](https://progress-bar.dev/96)
### Final Score
Final_score = 79.5